Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames

ABSTRACT

To acquire a high-resolution frame from a plurality of frames sampled from a video image, it is necessary to obtain a high-resolution frame with reduced picture quality degradation regardless of motion of a subject included in the frame. Because of this, between a plurality of contiguous frames Fr N  and Fr N+1 , there is estimated a correspondent relationship. Based on the correspondent relationship, the frames Fr N+1  and Fr N  are interposed to obtain first and second interpolated frames Fr H1  and Fr H2 . Based on the correspondent relationship, the coordinates of the frame Fr N+1  are transformed, and from a correlation value with the frame Fr N , there is obtained a weighting coefficient α(x ∘ , y ∘ ) that makes the weight of the first interpolated frame Fr H1  greater as a correlation becomes greater. With the weighting coefficient, the first and second interpolated frames are weighted and added to acquire a synthesized frame Fr G .

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a divisional of U.S. application Ser. No. 10/646,753, filed Aug.25, 2003, which claims priority from Japanese Patent Applications Nos.2002-249212 filed on Aug. 28, 2002, 2002-249213 filed on Aug. 28, 2002,2002-284126 filed on Sep. 27, 2002, 2002-284127 filed Sep. 27, 2002 and2002-284128 filed Sep. 27, 2002. The entire disclosures of the priorapplications are incorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a video image synthesis method and avideo image synthesizer for synthesizing a plurality of contiguousframes sampled from a video image to acquire a synthesized frame whoseresolution is higher than the sampled frame, and a program for causing acomputer to execute the synthesis method.

The present invention also relates to an image processing method andimage processor for performing image processing on one frame sampledfrom a video image to acquire a processed frame, and a program forcausing a computer to execute the processing method.

2. Description of the Related Art

With the recent spread of digital video cameras, it is becoming possibleto handle a video image in units of single frames. When printing such avideo image frame, the resolution of the frame needs to be made high toenhance the picture quality. Because of this, there has been disclosed amethod of sampling a plurality of frames from a video image andacquiring one synthesized frame whose resolution is higher than thesampled frames (e.g., Japanese Unexamined Patent Publication No.2000-354244). This method obtains a motion vector among a plurality offrames, and computes a signal value that is interpolated between pixels,when acquiring a synthesized frame from a plurality of frames, based onthe motion vector. Particularly, the method disclosed in theaforementioned publication No. 2000-354244 partitions each frame into aplurality of blocks, computes an orthogonal coordinate coefficient forblocks corresponding between frames, and synthesizes information about ahigh-frequency wave in this orthogonal coordinate coefficient andinformation about a low-frequency wave in another block to compute apixel value that is interpolated. Therefore, a synthesized frame withhigh picture quality can be obtained without reducing the requiredinformation. Also, in this method, the motion vector is computed withresolution finer than a distance between pixels, so a synthesized frameof high picture quality can be obtained by accurately compensating forthe motion between frames.

When synthesizing a plurality of video image frames, it is alsonecessary to acquire correspondent relationships between pixels of theframes in a motion area. The correspondent relationship is generallyobtained by employing block matching methods or differential(spatio-temporal gradient) methods. However, since the block matchingmethods are based on the assumption that a moved quantity within a blockis in the same direction, the methods are lacking in flexibility withrespect to various motions such as rotation, enlargement, reduction, anddeformation. Besides, these methods have the disadvantage that they aretime-consuming and impractical. On the other hand, the gradient methodshave the disadvantage that they cannot obtain stable solutions, comparedwith block matching methods. There is a method for overcoming thesedisadvantages (see, for example, Yuji Nakazawa, Takashi Komatsu, andTakahiro Saito, “Acquisition of High-Definition Digital Images byInterframe Synthesis,” Television Society Journal, 1995, Vol. 49, No. 3,pp. 299-308). This method employs one sampled frame as a referenceframe, places a reference patch consisting of one or a plurality ofrectangular areas on the reference frame, and respectively placespatches which are the same as the reference patch, on the others of thesampled frames. The patches are moved and/or deformed in the otherframes so that an image within each patch coincides with an image withinthe reference patch. Based on the patches after the movement and/ordeformation and on the reference patch, this method computes acorrespondent relationship between a pixel within the patch of each ofthe other frames and a pixel within the reference patch, therebysynthesizing a plurality of frames accurately.

The above-described method is capable of obtaining a synthesized frameof high definition by estimating a correspondent relationship betweenthe reference frame and the succeeding frame and then assigning thereference frame and the succeeding frame to a synthesized image that hasthe finally required resolution.

However, in the method disclosed by Nakazawa, et al., when the motion ofa subject in the succeeding frame is extremely great, or when a subjectlocally included in the succeeding frame moves complicatedly or at anextremely high speed, there are cases where the motion of a subjectcannot be followed by the movement and/or deformation of a patch. If themotion of a subject cannot be followed by the movement and/ordeformation of a patch, then a synthesized frame will become blurred asa whole or a subject with a great motion included in a frame will becomeblurred. As a result, the above-described method cannot obtain asynthesized frame of high picture quality.

Also, in the method disclosed by Nakazawa, et al., an operator manuallysets the range of frames that include a reference frame when sampling aplurality of frames from a video image, that is, the number of framesthat are used for acquiring a synthesized frame. Because of this, theoperator needs to have an expert knowledge of image processing, and thesetting of the number of frames will be time-consuming. Also, the manualsetting of the number of frames may vary according to each person'ssubjective point of view, so a suitable range of frames cannot always beobtained objectively. This has an adverse influence on the quality ofsynthesized frames.

Further, the method disclosed by Nakazawa, et al. selects one or aplurality of reference frames when sampling a plurality of frames from avideo image, and samples a predetermined range of frames for eachreference frame, including the reference frame. The selection ofreference frames is performed manually by an operator, so the operatormust have an expert knowledge of image processing and the selection istime-consuming. Also, the manual selection of reference frames may varyaccording to each person's subjective point of view, so proper referenceframes cannot always be determined objectively. This has an adverseinfluence on the quality of synthesized frames. In addition, referenceframes are set by the operator's judgement, so the intention of aphotographer cannot always be reflected and a synthesized frame withscenes desired by the photographer cannot be obtained.

Also, with the spread of digital video cameras, the video images takenby digital video cameras can be stored in a personal computer (PC), andthe video images can be freely edited or processed. Video image datarepresenting a video image can be downloaded into a PC by archiving thevideo image data in a database and accessing the database through anetwork from the PC. However, the amount of data for video image data islarge and the contents of the data cannot be recognized until it isplayed back, so it is difficult to handle, compared with still images.

To easily understand the contents of video images archived in a PC ordatabase, there has been proposed a method of detecting a frame thatrepresents a scene contained in a video image, and attaching this frameto the video image data (e.g., Japanese Unexamined Patent PublicationNo. 9 (1997)-233422). According to this method, the contents of a videoimage can be grasped by referring to a frame attached to video imagedata, so it becomes possible to handle the video image data easily.

However, in the video image, unlike still images, each frame on atemporal axis in the video image includes a blur unique to the videoimage. For instance, a subject in motion, which is included in a videoimage, has a blur proportional to the moved quantity in the movingdirection. Also, video images are low in resolution, compared to stillimages taken by digital still cameras, etc. Therefore, the picturequality of frames, sampled from a video image by the method disclosed inthe above-described Japanese Unexamined Patent Publication No.9(1997)-233422, are not so high.

SUMMARY OF THE INVENTION

The present invention has been made in view of the circumstancesdescribed above. Accordingly, it is a first object of the presentinvention to obtain a synthesized frame in which picture qualitydegradation has been reduced regardless of the motion of a subjectincluded in a frame. A second object of the present invention is todetermine a suitable range of frames easily and objectively and obtain asynthesized frame of good quality, when synthesizing a plurality offrames sampled from a video image. A third object of the presentinvention is to easily and objectively determine a proper referenceframe reflecting the intention of a photographer and obtain asynthesized frame of good quality, when synthesizing a plurality offrames sampled from a video image. A fourth object of the presentinvention is to obtain frames of high picture quality from a videoimage.

To achieve the objects of the present invention described above, thereis provided a first video image synthesis method. The first synthesismethod of the present invention comprises the steps of:

sampling two contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangularareas on one of the two frames which is used as a reference frame, thenplacing on the other of the two frames a second patch which is the sameas the reference patch, then moving and/or deforming the second patch inthe other frame so that an image within the second patch coincides withan image within the reference patch, and estimating a correspondentrelationship between a pixel within the second patch on the other frameand a pixel within the reference patch on the reference frame, based onthe second patch after the movement and/or deformation and on thereference patch;

acquiring a first interpolated frame whose resolution is higher thaneach of the frames, by performing interpolation either on the imagewithin the second patch of the other frame or on the image within thesecond patch of the other frame and image within the reference patch ofthe reference frame, based on the correspondent relationship;

acquiring a second interpolated frame whose resolution is higher thaneach of the frames, by performing interpolation on the image within thereference patch of the reference frame;

acquiring a coordinate-transformed frame by transforming coordinates ofthe image within the second patch of the other frame to a coordinatespace of the reference frame, based on the correspondent relationship;

computing a correlation value that represents a correlation between theimage within the patch of the coordinate-transformed frame and the imagewithin the reference patch of the reference frame; acquiring a weightingcoefficient that makes a weight of the first interpolated frame greateras the correlation becomes greater, when synthesizing the firstinterpolated frame and second interpolated frame, based on thecorrelation value; and

acquiring a synthesized frame by weighting and synthesizing the firstand second interpolated frames, based on the weighting coefficient.

The aforementioned correlation value may be computed betweencorresponding pixels of the images within the reference patch of thereference frame and within the patch of the coordinate-transformedframe, but it may also be computed between corresponding local areas,rectangular areas of patches, or frames. In this case, theaforementioned weighting coefficient is likewise acquired for eachpixel, each local area, each rectangular area, or each frame.

In accordance with the present invention, there is provided a secondvideo image synthesis method. The second synthesis method of the presentinvention comprises the steps of:

sampling three or more contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangularareas on one of the three or more frames which is used as a referenceframe, then respectively placing on the others of the three or moreframes patches which are the same as the reference patch, then movingand/or deforming the patches in the other frames so that an image withinthe patch of each of the other frames coincides with an image within thereference patch, and respectively estimating correspondent relationshipsbetween pixels within the patches of the other frames and a pixel withinthe reference patch of the reference frame, based on the patches of theother frames after the movement and/or deformation and on the referencepatch;

acquiring a plurality of first interpolated frames whose resolution ishigher than each of the frames, by performing interpolation either onthe image within the patch of each of the other frames or on the imagewithin the patch of each of the other frames and image within thereference patch of the reference frame, based on the correspondentrelationships;

acquiring one or a plurality of second interpolated frames whoseresolution is higher than each of the frames and which are correlatedwith the plurality of first interpolated frames, by performinginterpolation on the image within the reference patch of the referenceframe;

acquiring a plurality of coordinate-transformed frames by transformingcoordinates of the images within the patches of the other frames to acoordinate space of the reference frame, based on the correspondentrelationships;

computing correlation values that represent a correlation between theimage within the patch of each of the coordinate-transformed frames andthe image within the reference patch of the reference frame;

acquiring weighting coefficients that make a weight of the firstinterpolated frame greater as the correlation becomes greater, whensynthesizing the first interpolated frame and second interpolated frame,based on the correlation values; and

acquiring intermediate synthesized frames by weighting and synthesizingthe first and second interpolated frames that correspond to each otheron the basis of the weighting coefficients, and acquiring a synthesizedframe by synthesizing the intermediate synthesized frames.

In the second synthesis method of the present invention, while aplurality of correlation values are computed between the reference frameand other frames, the average or median value of the correlation valuesmay be employed for acquiring the aforementioned weighting coefficient.

The expression “acquiring a plurality of second interpolated frameswhich are correlated with the plurality of first interpolated frames” isintended to mean acquiring a number of second interpolated framescorresponding to the number of first interpolated frames. That is, apixel value within a reference patch is interpolated so that it isassigned at the same pixel position as a pixel position in a firstinterpolated frame that has a pixel value, whereby a second interpolatedframe corresponding to that first interpolated frame is acquired. Thisprocessing is performed on all of the first interpolated frames.

On the other hand, the expression “acquiring one second interpolatedframe which is correlated with the plurality of first interpolatedframes” is intended to mean acquiring one second interpolated frame.That is, a pixel value within a reference patch is interpolated so thatit is assigned at a predetermined pixel position in a secondinterpolated frame such as an integer pixel position, regardless of apixel position in a first interpolated frame that has a pixel value. Inthis manner, one second interpolated frame is acquired. In this case, apixel value at each of the pixel positions in a plurality of firstinterpolated frames, and a pixel value at a predetermined pixel positionin a second interpolated frame closest to that pixel value, are causedto correspond to each other.

According to the present invention, a plurality of contiguous frames arefirst sampled from a video image. Then, a reference patch comprising oneor a plurality of rectangular areas is placed on one of the frames,which is used as a reference frame. Next, a second patch that is thesame as the reference patch is placed on the other of the frames. Thesecond patch in the other frame is moved and/or deformed so that animage within the second patch coincides with an image within thereference patch. Based on the second patch after the movement and/ordeformation and on the reference patch, there is estimated acorrespondent relationship between a pixel within the second patch onthe other frame and a pixel within the reference patch on the referenceframe.

By performing interpolation either on the image within the second patchof the other frame or on the image within the second patch of the otherframe and the image within the reference patch of the reference frame,based on the correspondent relationship, there is acquired a firstinterpolated frame whose resolution is higher than each of the frames.Note that in the case where three or more frames are sampled, there areacquired a plurality of first interpolated frames. When the motion of asubject in each frame is small, the first interpolated frame representsa high-definition image whose resolution is higher than each frame. Onthe other hand, when the motion of a subject in each frame is great orcomplicated, a moving subject in the first interpolated frame becomesblurred.

In addition, by interpolating an image within the reference patch of thereference frame, there is obtained a second interpolated frame whoseresolution is higher than each frame. In the case where three or moreframes are sampled, one or a plurality of second interpolated frames areacquired with respect to a plurality of first interpolated frames. Thesecond interpolated frame is obtained by interpolating only one frame,so it is inferior in definition to the first interpolated frame, buteven when the motion of a subject is great or complicated, it does notbecome as blurred.

Moreover, the coordinate-transformed frame is acquired by transformingthe coordinates of the image within the second patch of the other frameto a coordinate space of the reference frame, based on the correspondentrelationship. The correlation value is computed and represents acorrelation between the image within the patch of thecoordinate-transformed frame and the image within the reference patch ofthe reference frame. The weighting coefficient, which is employed whensynthesizing the first interpolated frame and the second interpolatedframe, is computed based on the correlation value. As the correlationbetween the coordinate-transformed frame and the reference frame becomesgreater, the weighting coefficient makes the weight of the firstinterpolated frame greater. In the case where three or more frames aresampled, the coordinate-transformed frame, correlation value, andweighting coefficient are acquired for each of the frames other than thereference frame.

If the motion of a subject in each frame is small, the correlationbetween the coordinate-transformed frame and the reference frame becomesgreat, but if the motion is great or complicated, the correlationbecomes small. Therefore, by weighting and synthesizing the firstinterpolated frame and second interpolated frame on the basis of theweighting coefficient computed by the weight computation means, when themotion of a subject is small there is obtained a synthesized frame inwhich the ratio of the first interpolated frame with high definition ishigh, and when the motion is great there is obtained a synthesized frameincluding at a high ratio the second interpolated frame in which theblurring of a moving subject has been reduced. In the case where threeor more frames are sampled, first and second interpolated framescorresponding to each other are synthesized to acquire intermediatesynthesized frames. The intermediate synthesized frames are furthercombined into a synthesized frame.

Therefore, in the case where the motion of a subject in each frame isgreat, the blurring of a subject in the synthesized frame is reduced,and when the motion is small, high definition is obtained. In thismanner, a synthesized frame with high picture quality can be obtainedregardless of the motion of a subject included in each frame.

In the above-described synthesis methods of the present invention, whenthe aforementioned correlation value has been computed for each of thepixels and/or each of the local regions that constitute each of theframes, the aforementioned correlation value may be filtered to computea filtered correlation value, and the weighting coefficient may beacquired based on the filtered correlation value.

In this case, when the aforementioned correlation value has beencomputed for each of the pixels and/or each of the local regions thatconstitute each of the frames, the correlation value is filtered tocompute a filtered correlation value, and the weighting coefficient isacquired based on the filtered correlation value. Because of this, achange in the weighting coefficient in the coordinate space of a framebecomes smooth, and consequently, image changes in areas wherecorrelation values change can be smoothed. This is able to give thesynthesized frame a natural look.

The expression “the correlation value is filtered” is intended to meanthat a change in the correlation value is smoothed. More specifically,low-pass filters, median filters, maximum value filters, minimum valuefilters, etc., can be employed

In the first and second synthesis methods of the present invention, whenthe aforementioned correlation value has been computed for each of thepixels and/or each of the local regions that constitute each of theframes, the aforementioned weighting coefficient maybe interpolated toacquire weighting coefficients for all pixels that constitute the firstand second interpolated frames.

That is, the number of pixels in the first and second interpolatedframes becomes greater than that of each frame by interpolation, but theweighting coefficient is computed for only the pixels of sampled frames.Because of this, by interpolating the weighting coefficients acquiredfor the neighboring pixels, weighing coefficients for the increasedpixels may be computed. Also, the pixels increased by interpolation maybe weighted and synthesized, employing the weighting coefficientsacquired for the pixels that are originally present around the increasedpixels.

In this case, when the aforementioned correlation value has beencomputed for each of the pixels and/or each of the local regions thatconstitute each of the frames, the aforementioned weighting coefficientare interpolated to acquire weighting coefficients for all pixels thatconstitute the first and second interpolated frames. Therefore, sincethe pixels increased by interpolation are also weighted and synthesizedby the weighting coefficients acquired for those pixels, an image canchange naturally in local areas where correlation values change.

In the first and second synthesis methods of the present invention, theaforementioned weighting coefficient may be acquired by referring to anonlinear graph in which the aforementioned correlation value isrepresented in the horizontal axis and the aforementioned weightingcoefficient in the vertical axis.

In this case, the aforementioned weighting coefficient is acquired byreferring to the nonlinear graph in which the aforementioned correlationvalue is represented in the horizontal axis and the aforementionedweighting coefficient in the vertical axis. This can give a synthesizedframe a natural look in local areas where correlation values change.

It is preferable that the nonlinear graph employ a graph in which valueschange smoothly and slowly at boundary portions, in the case that acorrelation value is represented in the horizontal axis and a weightingcoefficient in the vertical axis.

In the first and second synthesis methods of the present invention, theaforementioned estimation of the correspondent relationship, acquisitionof the first interpolated frame, acquisition of the second interpolatedframe, acquisition of the coordinate-transformed frame, computation ofthe correlation value, acquisition of the weighting coefficient, andacquisition of the synthesized frame may be performed by employing atleast one component that constitutes the aforementioned frame.

In this case, the aforementioned estimation of the correspondentrelationship, acquisition of the first interpolated frame, acquisitionof the second interpolated frame, acquisition of thecoordinate-transformed frame, computation of the correlation value,acquisition of the weighting coefficient, and acquisition of thesynthesized frame are performed, employing at least one component thatconstitutes the aforementioned frame. Therefore, the first and secondsynthesis methods of the present invention are capable of obtaining asynthesized frame in which picture quality degradation has been reducedfor each component, and obtaining a synthesized frame of high picturequality consisting of frames synthesized for each component.

The expression “at least one component that constitutes the frame” isintended to mean, for example, at least one of RGB (red, green, andblue) components, at least one of YCC (luminance and color difference)components, etc. In the case where a frame consists of YCC components,the luminance component is preferred.

In accordance with the present invention, there is provided a firstvideo image synthesizer. The first synthesizer of the present inventioncomprises:

sampling means for sampling two contiguous frames from a video image;

correspondent relationship estimation means for placing a referencepatch comprising one or a plurality of rectangular areas on one of thetwo frames which is used as a reference frame, then placing on the otherof the two frames a second patch which is the same as the referencepatch, then moving and/or deforming the second patch in the other frameso that an image within the second patch coincides with an image withinthe reference patch, and estimating a correspondent relationship betweena pixel within the second patch on the other frame and a pixel withinthe reference patch on the reference frame, based on the second patchafter the movement and/or deformation and on the reference patch;

first interpolation means for acquiring a first interpolated frame whoseresolution is higher than each of the frames, by performinginterpolation either on the image within the second patch of the otherframe or on the image within the second patch of the other frame andimage within the reference patch of the reference frame, based on thecorrespondent relationship;

second interpolation means for acquiring a second interpolated framewhose resolution is higher than each of the frames, by performinginterpolation on the image within the reference patch of the referenceframe;

coordinate transformation means for acquiring a coordinate-transformedframe by transforming coordinates of the image within the second patchof the other frame to a coordinate space of the reference frame, basedon the correspondent relationship;

correlation-value computation means for computing a correlation valuethat represents a correlation between the image within the patch of thecoordinate-transformed frame and the image within the reference patch ofthe reference frame;

weighting-coefficient acquisition means for acquiring a weightingcoefficient that makes a weight of the first interpolated frame greateras the correlation becomes greater, when synthesizing the firstinterpolated frame and second interpolated frame, based on thecorrelation value; and

synthesis means for acquiring a synthesized frame by weighting andsynthesizing the first and second interpolated frames, based on theweighting coefficient.

In accordance with the present invention, there is provided asecondvideo image synthesizer. The secondvideo image synthesizer of thepresent invention comprises:

sampling means for sampling three or more contiguous frames from a videoimage;

correspondent relationship estimation means for placing a referencepatch comprising one or a plurality of rectangular areas on one of thethree or more frames which is used as a reference frame, thenrespectively placing on the others of the three or more frames patcheswhich are the same as the reference patch, then moving and/or deformingthe patches in the other frames so that an image within the patch ofeach of the other frames coincides with an image within the referencepatch, and respectively estimating correspondent relationships betweenpixels within the patches of the other frames and a pixel within thereference patch of the reference frame, based on the patches of theother frames after the movement and/or deformation and on the referencepatch;

first interpolation means for acquiring a plurality of firstinterpolated frames whose resolution is higher than each of the frames,by performing interpolation either on the image within the patch of eachof the other frames or on the image within the patch of each of theother frames and image within the reference patch of the referenceframe, based on the correspondent relationships;

second interpolation means for acquiring one or a plurality of secondinterpolated frames whose resolution is higher than each of the framesand which are correlated with the plurality of first interpolatedframes, by performing interpolation on the image within the referencepatch of the reference frame;

coordinate transformation means for acquiring a plurality ofcoordinate-transformed frames by transforming coordinates of the imageswithin the patches of the other frames to a coordinate space of thereference frame, based on the correspondent relationships;

correlation-value computation means for computing correlation valuesthat represent a correlation between the image within the patch of eachof the coordinate-transformed frames and the image within the referencepatch of the reference frame;

weighting-coefficient acquisition means for acquiring weightingcoefficients that make a weight of the first interpolated frame greateras the correlation becomes greater, when synthesizing the firstinterpolated frame and second interpolated frame, based on thecorrelation values; and

synthesis means for acquiring intermediate synthesized frames byweighting and synthesizing the first and second interpolated frames thatcorrespond to each other on the basis of the weighting coefficients, andacquiring a synthesized frame by synthesizing the intermediatesynthesized frames.

In the first and second video image synthesizers of the presentinvention, when the aforementioned correlation value has been computedfor each the of pixels and/or each of the local regions that constituteeach of the frames, the synthesizer may further comprise means forfiltering the correlation value to compute a filtered correlation value,and the aforementioned weighting-coefficient acquisition means mayacquire the weighting coefficient, based on the filtered correlationvalue.

In the first and second video image synthesizers of the presentinvention, when the aforementioned correlation value has been computedfor each of the pixels and/or each of the local regions that constituteeach of the frames, the aforementioned weighting-coefficient acquisitionmeans may perform interpolation on the weighting coefficient, therebyacquiring weighting coefficients for all pixels that constitute thefirst and second interpolated frames.

In the first and second video image synthesizers of the presentinvention, the aforementioned weighting-coefficient acquisition meansmay acquire the weighting coefficient by referring to a nonlinear graphin which the correlation value is represented in the horizontal axis andthe weighting coefficient in the vertical axis.

In the first and second video image synthesizers of the presentinvention, the correspondent relationship estimation means, the firstinterpolation means, the second interpolation means, the coordinatetransformation means, the correlation-value computation means, theweighting-coefficient acquisition means, and the synthesis means mayperform the estimation of the correspondent relationship, acquisition ofthe first interpolated frame, acquisition of the second interpolatedframe, acquisition of the coordinate-transformed frame, computation ofthe correlation value, acquisition of the weighting coefficient, andacquisition of the synthesized frame, by employing at least onecomponent that constitutes the aforementioned frame.

Note that the first and second synthesis methods of the presentinvention may be provided as programs to be executed by a computer.

In accordance with the present invention, there is provided a thirdvideo image synthesis method. The third synthesis method of the presentinvention comprises the steps of:

sampling two contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangularareas on one of the two frames which is used as a reference frame, thenplacing on the other of the two frames a second patch which is the sameas the reference patch, then moving and/or deforming the second patch inthe other frame so that an image within the second patch coincides withan image within the reference patch, and estimating a correspondentrelationship between a pixel within the second patch on the other frameand a pixel within the reference patch on the reference frame, based onthe second patch after the movement and/or deformation and on thereference patch;

acquiring a first interpolated frame whose resolution is higher thaneach of the frames, by performing interpolation either on the imagewithin the second patch of the other frame or on the image within thesecond patch of the other frame and image within the reference patch ofthe reference frame, based on the correspondent relationship;

acquiring a second interpolated frame whose resolution is higher thaneach of the frames, by performing interpolation on the image within thereference patch of the reference frame;

acquiring edge information that represents an edge intensity of theimage within the reference patch of the reference frame and/or imagewithin the patch of the other frame;

acquiring a weighting coefficient that makes a weight of the firstinterpolated frame greater as the edge information becomes greater, whensynthesizing the first interpolated frame and second interpolated frame,based on the edge information; and

acquiring a synthesized frame by weighting and synthesizing the firstand second interpolated frames, based on the weighting coefficient.

In accordance with the present invention, there is provided a fourthvideo image synthesis method. The fourth synthesis method of the presentinvention comprises the steps of:

sampling three or more contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangularareas on one of the three or more frames which is used as a referenceframe, then respectively placing on the others of the three or moreframes patches which are the same as the reference patch, then movingand/or deforming the patches in the other frames so that an image withinthe patch of each of the other frames coincides with an image within thereference patch, and respectively estimating correspondent relationshipsbetween pixels within the patches of the other frames and a pixel withinthe reference patch of the reference frame, based on the patches of theother frames after the movement and/or deformation and on the referencepatch;

acquiring a plurality of first interpolated frames whose resolution ishigher than each of the frames, by performing interpolation either onthe image within the patch of each of the other frames or on the imagewithin the patch of each of the other frames and image within thereference patch of the reference frame, based on the correspondentrelationships;

acquiring one or a plurality of second interpolated frames whoseresolution is higher than each of the frames and which are correlatedwith the plurality of first interpolated frames, by performinginterpolation on the image within the reference patch of the referenceframe;

acquiring edge information that represents an edge intensity of theimage within the reference patch of the reference frame and/or imagewithin the patch of each of the other frames;

acquiring weighting coefficients that make a weight of the firstinterpolated frame greater as the edge information becomes greater, whensynthesizing the first interpolated frame and second interpolated frame,based on the edge information; and

acquiring intermediate synthesized frames by weighting and synthesizingthe first and second interpolated frames that correspond to each otheron the basis of the weighting coefficients, and acquiring a synthesizedframe by synthesizing the intermediate synthesized frames.

In the fourth synthesis method of the present invention, while manypieces of edge information representing the edge intensity of an imagewithin the patch of each of the other frames are obtained between thereference frame and the other frames, the average or median value of theedge intensities maybe obtained as edge information that is employed foracquiring the aforementioned weighting coefficient.

The expression “acquiring a plurality of second interpolated frameswhich are correlated with the plurality of first interpolated frames” isintended to mean acquiring a number of second interpolated framescorresponding to the number of first interpolated frames. That is, apixel value within a reference patch is interpolated so that it isassigned at the same pixel position as a pixel position in a firstinterpolated frame that has a pixel value, whereby a second interpolatedframe corresponding to that first interpolated frame is acquired. Thisprocessing is performed on all of the first interpolated frames.

On the other hand, the expression “acquiring one second interpolatedframe which is correlated with the plurality of first interpolatedframes” is intended to mean acquiring one second interpolated frame.That is, a pixel value within a reference patch is interpolated so thatit is assigned at a predetermined pixel position in a secondinterpolated frame such as an integer pixel position, regardless of apixel position in a first interpolated frame that has a pixel value. Inthis manner, one second interpolated frame is acquired. In this case, apixel value at each of the pixel positions in a plurality of firstinterpolated frames, and a pixel value at a predetermined pixel positionin a second interpolated frame closest to that pixel value, are causedto correspond to each other.

According to the present invention, a plurality of contiguous frames arefirst sampled from a video image. Then, a reference patch comprising oneor a plurality of rectangular areas is placed on one of the frames,which is used as a reference frame. Next, a second patch that is thesame as the reference patch is placed on the other of the frames. Thesecond patch in the other frame is moved and/or deformed so that animage within the second patch coincides with an image within thereference patch. Based on the second patch after the movement and/ordeformation and on the reference patch, there is estimated acorrespondent relationship between a pixel within the second patch onthe other frame and a pixel within the reference patch on the referenceframe.

By performing interpolation either on the image within the second patchof the other frame or on the image within the second patch of the otherframe and the image within the reference patch of the reference frame,based on the correspondent relationship, there is acquired a firstinterpolated frame whose resolution is higher than each of the frames.Note that in the case where three or more frames are sampled, there areacquired a plurality of first interpolated frames. When the motion of asubject in each frame is small, the first interpolated frame representsa high-definition image whose resolution is higher than each frame. Onthe other hand, when the motion of a subject in each frame is great orcomplicated, a moving subject in the first interpolated frame becomesblurred.

In addition, by interpolating an image within the reference patch of thereference frame, there is obtained a second interpolated frame whoseresolution is higher than each frame. In the case where three or moreframes are sampled, one or a plurality of second interpolated frames areacquired with respect to a plurality of first interpolated frames. Thesecond interpolated frame is obtained by interpolating only one frame,so it is inferior in definition to the first interpolated frame, buteven when the motion of a subject is great or complicated, it does notbecome as blurred.

Moreover, there is obtained edge information that represents an edgeintensity of the image within the reference patch of the reference frameand/or image within the patch of the other frame. Based on the edgeinformation, there is computed a weighting coefficient that is employedin synthesizing the first interpolated frame and the second interpolatedframe. As the edge intensity represented by the edge information becomesgreater, the weighting coefficient makes the weight of the firstinterpolated frame greater.

If the motion of a subject in each frame is small, the edge intensity ofthe reference frame and/or the other frame becomes great, but if themotion is great or complicated, it moves the contour of the subject andmakes the edge intensity small. Therefore, by weighting and synthesizingthe first interpolated frame and second interpolated frame on the basisof the weighting coefficient computed by the weight computation means,when the motion of a subject is small there is obtained a synthesizedframe in which the ratio of the first interpolated frame with highdefinition is high, and when the motion is great there is obtained asynthesized frame including at a high ratio the second interpolatedframe in which the blurring of a moving subject has been reduced. In thecase where three or more frames are sampled, first and secondinterpolated frames corresponding to each other are synthesized toacquire intermediate synthesized frames. The intermediate synthesizedframes are further combined into a synthesized frame.

Therefore, in the case where the motion of a subject in each frame isgreat, the blurring of a subject in the synthesized frame is reduced,and when the motion is small, high definition is obtained. In thismanner, a synthesized frame with high picture quality can be obtainedregardless of the motion of a subject included in each frame.

In the third and fourth synthesis methods of the present invention, whenthe edge information has been computed for each of the pixels thatconstitute each of the frames, the aforementioned weighting coefficientmay be interpolated to acquire weighting coefficients for all pixelsthat constitute the first and second interpolated frames.

That is, the number of pixels in the first and second interpolatedframes becomes greater than that of each frame by interpolation, but theweighting coefficient is computed for only the pixels of sampled frames.Because of this, by interpolating the weighting coefficients acquiredfor the neighboring pixels, weighing coefficients for the increasedpixels may be computed. Also, the pixels increased by interpolation maybe weighted and synthesized, employing the weighting coefficientsacquired for the pixels that are originally present around the increasedpixels.

In this case, when the aforementioned edge information has been computedfor each of the pixels that constitute each of the frames, theaforementioned weighting coefficient are interpolated to acquireweighting coefficients for all pixels that constitute the first andsecond interpolated frames. Therefore, since the pixels increased byinterpolation are also weighted and synthesized by the weightingcoefficients acquired for those pixels, an image can change naturally inlocal areas where edge information changes.

In the third and fourth synthesis methods of the present invention, theestimation of the correspondent relationship, acquisition of the firstinterpolated frame, acquisition of the second interpolated frame,acquisition of the edge information, acquisition of the weightingcoefficient, and acquisition of the synthesized frame may be performedby employing at least one component that constitutes the frame.

In this case, the aforementioned estimation of the correspondentrelationship, acquisition of the first interpolated frame, acquisitionof the second interpolated frame, acquisition of thecoordinate-transformed frame, computation of the correlation value,acquisition of the weighting coefficient, and acquisition of thesynthesized frame are performed, employing at least one component thatconstitutes the aforementioned frame. Therefore, the third and fourthsynthesis methods of the present invention are capable of obtaining asynthesized frame in which picture quality degradation has been reducedfor each component, and obtaining a synthesized frame of high picturequality consisting of frames synthesized for each component.

The expression “at least one component that constitutes the frame” isintended to mean, for example, at least one of RGB (red, green, andblue) components, at least one of YCC (luminance and color difference)components, etc. In the case where a frame consists of YCC components,the luminance component is preferred.

In accordance with the present invention, there is provided a thirdvideo image synthesizer. The third video image synthesizer of thepresent invention comprises:

sampling means for sampling two contiguous frames from a video image;

correspondent relationship estimation means for placing a referencepatch comprising one or a plurality of rectangular areas on one of thetwo frames which is used as a reference frame, then placing on the otherof the two frames a second patch which is the same as the referencepatch, then moving and/or deforming the second patch in the other frameso that an image within the second patch coincides with an image withinthe reference patch, and estimating a correspondent relationship betweena pixel within the second patch on the other frame and a pixel withinthe reference patch on the reference frame, based on the second patchafter the movement and/or deformation and on the reference patch;

first interpolation means for acquiring a first interpolated frame whoseresolution is higher than each of the frames, by performinginterpolation either on the image within the second patch of the otherframe or on the image within the second patch of the other frame andimage within the reference patch of the reference frame, based on thecorrespondent relationship;

second interpolation means for acquiring a second interpolated framewhose resolution is higher than each of the frames, by performinginterpolation on the image within the reference patch of the referenceframe;

edge information acquisition means for acquiring edge information thatrepresents an edge intensity of the image within the reference patch ofthe reference frame and/or image within the patch of the other frame;

weighting-coefficient acquisition means for acquiring a weightingcoefficient that makes a weight of the first interpolated frame greateras the edge information becomes greater, when synthesizing the firstinterpolated frame and second interpolated frame, based on the edgeinformation; and

synthesis means for acquiring a synthesized frame by weighting andsynthesizing the first and second interpolated frames, based on theweighting coefficient.

In accordance with the present invention, there is provided a fourthvideo image synthesizer. The fourth video image synthesizer of thepresent invention comprises:

sampling means for sampling three or more contiguous frames from a videoimage;

correspondent relationship estimation means for placing a referencepatch comprising one or a plurality of rectangular areas on one of thethree or more frames which is used as a reference frame, thenrespectively placing on the others of the three or more frames patcheswhich are the same as the reference patch, then moving and/or deformingthe patches in the other frames so that an image within the patch ofeach of the other frames coincides with an image within the referencepatch, and respectively estimating correspondent relationships betweenpixels within the patches of the other frames and a pixel within thereference patch of the reference frame, based on the patches of theother frames after the movement and/or deformation and on the referencepatch;

first interpolation means for acquiring a plurality of firstinterpolated frames whose resolution is higher than each of the frames,by performing interpolation either on the image within the patch of eachof the other frames or on the image within the patch of each of theother frames and image within the reference patch of the referenceframe, based on the correspondent relationships;

second interpolation means for acquiring one or a plurality of secondinterpolated frames whose resolution is higher than each of the framesand which are correlated with the plurality of first interpolatedframes, by performing interpolation on the image within the referencepatch of the reference frame;

edge information acquisition means for acquiring edge information thatrepresents an edge intensity of the image within the reference patch ofthe reference frame and/or image within the patch of each of the otherframes;

weighting-coefficient acquisition means for acquiring weightingcoefficients that make a weight of the first interpolated frame greateras the edge information becomes greater, when synthesizing the firstinterpolated frame and second interpolated frame, based on the edgeinformation; and

synthesis means for acquiring intermediate synthesized frames byweighting and synthesizing the first and second interpolated frames thatcorrespond to each other on the basis of the weighting coefficients, andacquiring a synthesized frame by synthesizing the intermediatesynthesized frames.

In the third and fourth video image synthesizers of the presentinvention, when the aforementioned edge information has been computedfor each of the pixels that constitute each of the frames, theaforementioned weighting-coefficient acquisition means may performinterpolation on the weighting coefficient, thereby acquiring weightingcoefficients for all pixels that constitute the first and secondinterpolated frames.

In the third and fourth video image synthesizers of the presentinvention, the correspondent relationship estimation means, the firstinterpolation means, the second interpolation means, the edgeinformation acquisition means, the weighting-coefficient acquisitionmeans, and the synthesis means may perform the estimation of thecorrespondent relationship, acquisition of the first interpolated frame,acquisition of the second interpolated frame, acquisition of the edgeinformation, acquisition of the weighting coefficient, and acquisitionof the synthesized frame, by employing at least one component thatconstitutes the frame.

Note that the third and fourth synthesis methods of the presentinvention may be provided as programs to be executed by a computer.

In accordance with the present invention, there is provided a fifthvideo image synthesis method. The fifth synthesis method of the presentinvention comprises the steps of:

sampling a predetermined number of contiguous frames, which include areference frame and are two or more frames, from a video image;

placing a reference patch comprising one or a plurality of rectangularareas on the reference frame;

respectively placing patches which are the same as the reference patch,on the others of the predetermined number of frames;

moving and/or deforming the patches in the other frames so that an imagewithin the patch of each of the other frames approximately coincideswith an image within the reference patch;

respectively acquiring correspondent relationships between pixels withinthe patches of the other frames and a pixel within the reference patchof the reference frame, based on the patches of the other frames afterthe movement and/or deformation and on the reference patch; and

acquiring a synthesized frame from the predetermined number of frames,based on the correspondent relationships;

wherein the predetermined number of frames are determined based on imagecharacteristics of the video image or synthesized frame, and thepredetermined number of frames are sampled.

The image characteristics of a video image refer to characteristics thatcan have influence on the quality of a synthesized frame when acquiringthe frame from a video image. Examples are pixel sizes and resolution ofeach frame, frame rates, compression ratios, etc. The imagecharacteristics of a synthesized frame mean characteristics that canhave influence on the number of frames to be sampled or thedetermination of the required number of frames. Examples are pixel sizesand resolution of a synthesized frame, etc. Also, the magnificationratio of the pixel size of a synthesized frame to the pixel size of theframe of a video image is the image characteristics of a video image anda synthesized frame that can have an indirect influence on the qualityof synthesized frames.

In the fifth synthesis method of the present invention, the method ofacquiring the aforementioned image characteristics may be any type ofmethod if it can acquire the required image characteristics. Forinstance, for the image characteristics of a video image, attachedinformation, such as a tag attached to a video image, may be read, orvalues input by an operator may be employed. For the imagecharacteristics of a synthesized frame, values input by an operator maybe employed, or a fixed target value may be employed.

In a preferred form of the fifth synthesis method of the presentinvention, the aforementioned correspondent relationships are acquiredin order of the other frames closer to the reference frame, and acorrelation is acquired between each of the other frames, in which thecorrespondent relationship is acquired, and the reference frame. Whenthe correlation is lower than a predetermined threshold value,acquisition of the correspondent relationships is stopped, and thesynthesized frame is obtained based on the correspondent relationship byemploying the other frames, in which the correspondent relationship hasbeen acquired, and the reference frame.

When the reference frame is the first one or last one of the sampledframes, the expression “in order of the other frames closer to thereference frame” is intended to mean “in order of the other framesearlier in time series than the reference frame” or “in order of theother frames later in time series than the reference frame”. On theother hand, when the reference frame is not the first one or the lastone, the expression “in order of the other frames closer to thereference frame” is intended to mean both “in order of the other framesearlier in time series than the reference frame” and “in order of theother frames later in time series than the reference frame.”

In accordance with the present invention, there is provided a fifthvideo image synthesizer. The fifth video image synthesizer of thepresent invention comprises:

sampling means for sampling a predetermined number of contiguous frames,which include a reference frame and are two or more frames, from a videoimage;

correspondent relationship acquisition means for placing a referencepatch comprising one or a plurality of rectangular areas on thereference frame, then respectively placing on the others of thepredetermined number of frames patches which are the same as thereference patch, then moving and/or deforming the patches in the otherframes so that an image within the patch of each of the other framesapproximately coincides with an image within the reference patch, andrespectively acquiring correspondent relationships between pixels withinthe patches of the other frames and a pixel within the reference patchof the reference frame, based on the patches of the other frames afterthe movement and/or deformation and on the reference patch; and

frame synthesis means for acquiring a synthesized frame from thepredetermined number of frames, based on the correspondent relationshipsacquired by the correspondent relationship acquisition means;

wherein the sampling means is equipped with frame-number determinationmeans for determining the predetermined number of frames on the basis ofimage characteristics of the video image or synthesized frame, andsamples the predetermined number of frames determined by theframe-number determination means.

In a preferred form of the fifth video image synthesizer of the presentinvention, the correspondent relationship acquisition means acquires thecorrespondent relationships in order of other frames closer to thereference frame. Also, the fifth video image synthesizer furthercomprises stoppage means for acquiring a correlation between each of theother frames, in which the correspondent relationship is acquired by thecorrespondent relationship acquisition means, and the reference frame,and stopping a process which is being performed in the correspondentrelationship acquisition means when the correlation is lower than apredetermined threshold value. The frame synthesis means acquires thesynthesized frame by employing the other frames, in which thecorrespondent relationship has been acquired, and the reference frame,based on the correspondent relationship acquired by the correspondentrelationship acquisition means.

Note that the fifth synthesis method of the present invention may beprovided as a program to be executed by a computer.

According to the fifth video image synthesis method and synthesizer ofthe present invention, when sampling a plurality of contiguous framesfrom a video image and acquiring a synthesized frame, the number offrames to be sampled is determined based on the image characteristics ofthe video image and/or synthesized frame. Therefore, the operator doesnot need to sample frames manually, and the video image synthesis methodand synthesizer can be conveniently used. Also, by determining thenumber of frames on the basis of the image characteristics, a suitablenumber of frames can be objectively determined, so a synthesized framewith high quality can be obtained.

In the fifth video image synthesis method and synthesizer of the presentinvention, the frames of a determined number are sampled. Thecorrespondent relationship between a pixel within a reference patch onthe reference frame and a pixel within a patch on the succeeding frameis computed in order of other frames closer to the reference frame, andthe correlation between the reference frame and the succeeding frame isobtained. If the correlation is a predetermined threshold value orgreater, then a correspondent relationship with the next frame isacquired. On the other hand, if a frame whose correlation is less thanthe predetermined threshold value is detected, the acquisition ofcorrespondent relationships with other frames after the detected frameis stopped, even when the number of frames does not reach the determinedframe number. This can avoid acquiring a synthesized frame from areference frame and a frame whose correlation is low (e.g., a referenceframe for a scene and a frame for a switched scene), and makes itpossible to acquire a synthesized frame of higher quality.

In accordance with the present invention, there is provided a sixthvideo image synthesis method. The sixth synthesis method of the presentinvention comprises the steps of:

obtaining a contiguous frame group by detecting a plurality of framesthat represent contiguous scenes in a video image;

placing a reference patch comprising one or a plurality of rectangularareas on one of the plurality of frames included in the contiguous framegroup which is used as a reference frame;

respectively placing patches which are the same as the reference patch,on the others of the plurality of frames;

moving and/or deforming the patches in the other frames so that an imagewithin the patch of each of the other frames approximately coincideswith an image within the reference patch;

respectively acquiring correspondent relationships between pixels withinthe patches of the other frames and a pixel within the reference patchof the reference frame, based on the patches of the other frames afterthe movement and/or deformation and on the reference patch; and

acquiring a synthesized frame from the plurality of frames, based on thecorrespondent relationships.

The expression “contiguous scenes” is intended to mean scenes that haveapproximately the same contents in a video image. The expression“contiguous frame group” is intended to mean a plurality of frames thatconstitute one contiguous scene.

In the sixth synthesis method of the present invention, when detectingcontiguous frames, a correlation between adjacent frames, which isstarted from the reference frame, is acquired. The contiguous framegroup that is detected comprises frames ranging from the reference frameto a frame, which is closer to the reference frame, between a pair ofthe adjacent frames in which the correlation is lower than apredetermined first threshold value.

In the sixth synthesis method of the present invention, a histogram iscomputed for at least one of the Y, Cb, and Cr components of each of theadjacent frames (where the Y component is a luminance component and theCb and Cr components are color difference components). Also, a Euclideandistance for each component between the adjacent frames is computed byemploying the histogram. The sum of the Euclidean distances for thethree components is computed, and when the sum is a predetermined secondthreshold value or greater, the correlation between the adjacent framesis lower than the predetermined first threshold value.

The expression “at least one of the Y, Cb, and Cr components” isintended to mean one, two, or three of the luminance component and colordifference components. Preferred examples are only the luminancecomponent, or a combination of the three components.

In the sixth synthesis method of the present invention, theaforementioned histogram may be computed by dividing each of components,which are used, among the three components by a value greater than 1.

The sixth synthesis method of the present invention, as a method ofcomputing a correlation between adjacent frames, may compute adifference between pixel values of corresponding pixels of the adjacentframes for all corresponding pixels, and compute the sum of absolutevalues of the differences for all corresponding pixels. When the sum isa third threshold value or greater, the correlation between adjacentframes may be determined to be lower than the predetermined firstthreshold value.

In the sixth synthesis method of the present invention, theaforementioned correlation may be computed by employing a reduced imageor thinned image of each frame.

In a preferred form of the sixth synthesis method of the presentinvention, the detection of frames that constitute the contiguous framegroup is stopped when the number of detected frames reaches apredetermined upper limit value.

In accordance with the present invention, there is provided a sixthvideo imager synthesizer. The video image synthesizer of the presentinvention comprises:

contiguous frame group detection means for obtaining a contiguous framegroup by detecting a plurality of frames that represent contiguousscenes in a video image;

correspondent relationship acquisition means for placing a referencepatch comprising one or a plurality of rectangular areas on one of theplurality of frames included in the contiguous frame group which is usedas a reference frame, then respectively placing on the others of theplurality of frames patches which are the same as the reference patch,then moving and/or deforming the patches in the other frames so that animage within the patch of each of the other frames approximatelycoincides with an image within the reference patch, and respectivelyacquiring correspondent relationships between pixels within the patchesof the other frames and a pixel within the reference patch of thereference frame, based on the patches of the other frames after themovement and/or deformation and on the reference patch; and

frame synthesis means for acquiring a synthesized frame from theplurality of frames, based on the correspondent relationships acquiredby the correspondent relationship acquisition means.

In another preferred form of the sixth video image synthesizer of thepresent invention, the aforementioned correlation computation meanscomputes a histogram for at least one of the Y, Cb, and Cr components ofeach of the adjacent frames (where the Y component is a luminancecomponent and the Cb and Cr components are color difference components),also computes a Euclidean distance for each component between theadjacent frames by employing the histogram, and computes the sum of theEuclidean distances for the three components. When the sum is apredetermined second threshold value or greater, the aforementionedcontiguous frame group detection means judges that the correlationbetween the adjacent frames is lower than the predetermined firstthreshold value.

In another preferred form of the sixth video image synthesizer of thepresent invention, the aforementioned correlation computation meanscomputes a histogram for at least one of the Y, Cb, and Cr components ofeach of the adjacent frames (where the Y component is a luminancecomponent and the Cb and Cr components are color difference components),also computes a Euclidean distance for each component between theadjacent frames by employing the histogram, and computes the sum of theEuclidean distances for the three components. And when the sum is apredetermined second threshold value or greater, the aforementionedcontiguous frame group detection means judges that the correlationbetween the adjacent frames is lower than the predetermined firstthreshold value.

In the sixth video image synthesizer of the present invention, it isdesirable the correlation computation means compute the histogram bydividing each of components, which are used, among the three componentsby a value greater than 1 in order to achieve expedient processing.

In the sixth video image synthesizer of the present invention, theaforementioned correlation computation means may compute a differencebetween pixel values of corresponding pixels of the adjacent frames andalso compute the sum of absolute values of the differences for allcorresponding pixels. When the sum is a third threshold value orgreater, the contiguous frame group detection means may judge that thecorrelation between adjacent frames is lower than the predeterminedfirst threshold value.

It is desirable that to expedite processing, the aforementionedcorrelation computation means in the sixth video image synthesizer ofthe present invention compute the aforementioned correlation byemploying a reduced image or thinned image of each frame.

It is also desirable that the sixth video image synthesizer of thepresent invention further comprise stoppage means for stopping thedetection of frames, which constitute the contiguous frame group, whenthe number of frames detected by the contiguous frame group detectionmeans reaches a predetermined upper limit value.

Note that the sixth synthesis method of the present invention may beprovided as a program to be executed by a computer.

According to the sixth video image synthesis method and synthesizer ofthe present invention, the sampling means detects a plurality of framesrepresenting successive scenes as a contiguous frame group whenacquiring a synthesized frame from a video image, and acquires thesynthesized frame from this frame group. Therefore, an operator does notneed to sample frames manually, and the synthesis method and video imagesynthesizer can be conveniently used. In addition, a plurality of frameswithin each contiguous frame group represent scenes that haveapproximately the same contents, so the synthesis method and video imagesynthesizer are suitable for acquiring a synthesized frame of highquality.

In the sixth video image synthesis method and synthesizer of the presentinvention, there is provided a predetermined upper limit value. Indetecting a contiguous frame group, the detection of frames is stoppedwhen the number of frames in that contiguous frame group reaches thepredetermined upper limit value. This can avoid employing a great numberof frames wastefully when acquiring one synthesized frame, and rendersit possible to perform processing efficiently.

In accordance with the present invention, there is provided a seventhvideo image synthesis method. The seventh synthesis method of thepresent invention comprises the steps of:

extracting a frame group that constitutes one or more important scenesfrom a video image;

determining a frame, which is located at approximately a center, among aplurality of frames of the frame group as a reference frame for theimportant scene;

placing a reference patch comprising one or a plurality of rectangularareas on the reference frame;

respectively placing patches which are the same as the reference patch,on the others of the plurality of frames;

moving and/or deforming the patches in the other frames so that an imagewithin the patch of each of the other frames approximately coincideswith an image within the reference patch;

respectively acquiring correspondent relationships between pixels withinthe patches of the other frames and a pixel within the reference patchof the reference frame, based on the patches of the other frames afterthe movement and/or deformation and on the reference patch; and

acquiring a synthesized frame from the plurality of frames, based on thecorrespondent relationships.

The expression “important scene” is intended to mean a scene from whicha synthesized frame is obtained in a video image. For instance, whenrecording an image, there is a tendency to record an interesting scenefor a relatively long time (e.g., a few seconds) without moving acamera, so frames having approximately the same contents for arelatively long time can be considered to be an important scene inordinary video image data. On the other hand, in the case of a videoimage (security image) taken by a security camera, different scenes fora short time (e.g., scenes picking up an intruder), included in scenesof the same contents which continues for a long time, can be consideredimportant scenes.

In accordance with the present invention, there is provided an eighthvideo image synthesis method. The eighth synthesis method of the presentinvention comprises the steps of:

extracting a frame group that constitutes one or more important scenesfrom the video image;

extracting high-frequency components of each of a plurality of framesconstituting the frame group;

computing the sum of the high-frequency components for each of theframes;

determining a frame, in which the sum is highest, as a reference framefor the important scene;

placing a reference patch comprising one or a plurality of rectangularareas on the reference frame;

respectively placing patches which are the same as the reference patch,on the others of the plurality of frames;

moving and/or deforming the patches in the other frames so that an imagewithin the patch of each of the other frames approximately coincideswith an image within the reference patch;

respectively acquiring correspondent relationships between pixels withinthe patches of the other frames and a pixel within the reference patchof the reference frame, based on the patches of the other frames afterthe movement and/or deformation and on the reference patch; and

acquiring a synthesized frame from the plurality of frames, based on thecorrespondent relationships.

That is, the seventh synthesis method of the present inventiondetermines as a reference frame a frame, which is located atapproximately a center, among a plurality of frames of the extractedframe group. On the other hand, the eighth synthesis method of thepresent invention determines as a reference frame a frame, in which thesum of high-frequency components is highest, among the extracted frames.

In the seventh and eighth synthesis methods of the present invention,when extracting the aforementioned important scenes, a correlationbetween adjacent frames of the video image is computed. A set ofcontiguous frames where the correlation is high can be extracted as theframe group that constitutes one or more important scenes.

The expression “the correlation is high” is intended to mean that thecorrelation is higher than a predetermined threshold value. Thepredetermined threshold value may be set by an operator.

As a method of computing a correlation between adjacent frames, ahistogram is computed for the luminance component Y of each of theframes that constitute the aforementioned frame group. Using thehistogram, a Euclidean distance between adjacent frames is computed.When the Euclidean distance is smaller than a predetermined thresholdvalue, the correlation may be considered high. Also, a Euclideandistance for each component between the adjacent frames may be computedby employing the histogram. In this case, the sum of the Euclideandistances for the three components is computed, and when the sum issmaller than a predetermined threshold value, the correlation betweenthe adjacent frames may be considered high. Furthermore, a differencebetween the pixel values of corresponding pixels of adjacent frames maybe computed. In this case, the sum of the absolute values of thedifferences is computed, and when the sum is smaller than apredetermined threshold value, the correlation between the adjacentframes maybe considered high.

When extracting the aforementioned important scenes, the seventh andeighth synthesis methods of the present invention may compute acorrelation between adjacent frames of the video image; extract a set ofcontiguous frames where the correlation is high, as a frame group thatconstitutes temporary important scenes; respectively computecorrelations between the temporary important scenes not adjacent; andextract a frame group, interposed between two temporary important sceneswhere the correlation is high and which are closest to each other, asthe frame group that constitutes one or more important scenes.

The expression “correlation between the temporary important scenes” isintended to mean the correlation between frames that constitute theaforementioned temporary important scenes. Any type of correlation canbe employed if it can represent the correlation between the temporaryimportant scenes. For example, the correlations between the framesconstituting one of the two temporary important scenes and the framesconstituting the other of the two temporary important scenes arerespectively computed, and the sum of these correlations maybe employedas the correlation between two temporary important scenes. To shortenthe processing time, the correlation between the representative framesof frame groups respectively constituting two temporary important scenesmay be employed as the correlation between the two temporary importantscenes. The representative frame for the temporary important scenes maybe a frame that is located at approximately the center between thetemporary important scenes.

In accordance with the present invention, there is provided a seventhvideo image synthesizer. The seventh video image synthesizer of thepresent invention comprises:

important-scene extraction means for extracting a frame group thatconstitutes one or more important scenes from a video image;

reference-frame determination means for determining a frame, which islocated at approximately a center, among a plurality of frames of theframe group as a reference frame for the important scene;

correspondent relationship acquisition means for placing a referencepatch comprising one or a plurality of rectangular areas on thereference frame, then respectively placing on the others of theplurality of frames patches which are the same as the reference patch,then moving and/or deforming the patches in the other frames so that animage within the patch of each of the other frames approximatelycoincides with an image within the reference patch, and respectivelyacquiring correspondent relationships between pixels within the patchesof the other frames and a pixel within the reference patch of thereference frame, based on the patches of the other frames after themovement and/or deformation and on the reference patch; and

frame synthesis means for acquiring a synthesized frame from theplurality of frames, based on the correspondent relationships.

In accordance with the present invention, there is provided an eighthvideo image synthesizer. The eighth video image synthesizer of thepresent invention comprises:

important-scene extraction means for extracting a frame group thatconstitutes one or more important scenes from a video image;

reference-frame determination means for extracting high-frequencycomponents of each of a plurality of frames constituting the framegroup, then computing the sum of the high-frequency components for eachof the frames, and determining a frame, in which the sum is highest, asa reference frame for the important scene;

correspondent relationship acquisition means for placing a referencepatch comprising one or a plurality of rectangular areas on thereference frame, then respectively placing on the others of theplurality of frames patches which are the same as the reference patch,then moving and/or deforming the patches in the other frames so that animage within the patch of each of the other frames approximatelycoincides with an image within the reference patch, and respectivelyacquiring correspondent relationships between pixels within the patchesof the other frames and a pixel within the reference patch of thereference frame, based on the patches of the other frames after themovement and/or deformation and on the reference patch; and

frame synthesis means for acquiring a synthesized frame from theplurality of frames, based on the correspondent relationships.

In the seventh and eighth video image synthesizers of the presentinvention, the aforementioned important-scene extraction means isequipped with correlation computation means for computing a correlationbetween adjacent frames of the video image, and extracts a set ofcontiguous frames, in which the correlation computed by the correlationcomputation means is high, as the frame group that constitutes one ormore important scenes. Note that this important scene extraction meansis called first important scene extraction means.

In the seventh and eighth video image synthesizers of the presentinvention, the important-scene extraction means may comprise:

first correlation computation means for computing a correlation betweenadjacent frames of the video image;

temporary important scene extraction means for extracting a set ofcontiguous frames, in which the correlation computed by the firstcorrelation computation means is high, as a frame group that constitutestemporary important scenes; and

second correlation computation means for respectively computingcorrelations between the temporary important scenes not adjacent.

Also, the important-scene extraction means may extract a frame group,interposed between two temporary important scenes where the correlationcomputed by the second correlation commutation means is high and whichare closest to each other, as the frame group that constitutes one ormore important scenes.

Note that this important scene extraction means is called secondimportant scene extraction means.

In accordance with the present invention, there is provided a ninthvideo image synthesizer. The important-scene extraction means in theninth video image synthesizer of the present invention comprises thefirst important-scene extraction means of the seventh video imagesynthesizer and the second important-scene extraction means of theeighth video image synthesizer. The ninth video image synthesizerfurther includes selection means for selecting either the firstimportant-scene extraction means or the second important-sceneextraction means.

Note that the seventh and eighth synthesis methods of the presentinvention may be provided as programs to be executed by a computer.

According to the seventh and eighth synthesis methods of the presentinvention, the sampling means extracts frame groups constituting animportant scene from a video image, and determines the center frame of aplurality of frames constituting each frame group or a frame that ismost in focus, as the reference frame of the frame group. Therefore, theoperator does not need to set a reference frame manually, and theseventh and eighth video image synthesizer can be conveniently used. Insampling a plurality of frames, unlike a method of setting a referenceframe and then sampling frames in a range including the reference frame,frames constituting an important scene included in video image data areextracted and then a reference frame is determined so that a synthesizedframe is obtained for each important scene. Thus, the intention of anphotographer can be reflected.

In accordance with the present invention, there is provided a method ofacquiring a processed frame by performing image processing on a desiredframe sampled from a video image. The image processing method of thepresent invention comprises the steps of:

computing a similarity between the desired frame and at least one framewhich is temporally before and after the desired frame; and

acquiring the processed frame by obtaining a weighting coefficient thatbecomes greater if the similarity becomes greater, then weighting the atleast one frame with the weighting coefficient, and synthesizing theweighted frame and the desired frame.

The “synthesizing” can be performed, for example, by weighted addition.

To enhance picture quality when outputting some of the framesconstituting a video image as prints, Japanese Unexamined PatentPublication No. 2000-354244 discloses a method of sampling a pluralityof frames from a video image and acquiring a synthesized frame whoseresolution is higher than the sampled frames.

This method obtains a motion vector that represents the moving directionand moved quantity between one frame and another frame and, based on themotion vector, computes a signal value that is interpolated betweenpixels when synthesizing a high-resolution frame from a plurality offrames. Particularly, this method partitions each frame into a pluralityof blocks, computes an orthogonal coordinate coefficient for blockscorresponding between frames, and synthesizes information about ahigh-frequency wave in this orthogonal coordinate coefficient andinformation about a low-frequency wave in another block to compute apixel value that is interpolated. Therefore, this method is able toobtain a synthesized frame with high picture quality without reducingthe required information. Also, in this method, the motion vector iscomputed with resolution finer than a distance between pixels, so ahigh-frequency frame with higher picture quality can be obtained byaccurately compensating for the motion between frames.

The present invention may obtain processed image data by synthesizing atleast one frame and a desired frame by the method disclosed in theaforementioned publication No. 2000-354244.

In the image processing method of the present invention, the desiredframe may be partitioned into a plurality of areas. Also, the similaritymay be computed for each of corresponding areas in at least one framewhich correspond to the plurality of areas. The processed frame may beacquired by obtaining weighting coefficients that become greater if thesimilarity becomes greater, then weighting the corresponding areas ofthe at least one frame with the weighting coefficients, and synthesizingthe weighted areas and the plurality of areas.

In the image processing method of the present invention, the desiredframe maybe partitioned into a plurality of subject areas that areincluded in the desired frame; the similarity may be computed for eachof corresponding subject areas in at least one frame which correspond tothe plurality of subject areas; and the processed frame may be acquiredby obtaining weighting coefficients that become greater if thesimilarity becomes greater, then weighting the corresponding subjectareas of the at least one frame with the weighting coefficients, andsynthesizing the weighted subject areas and the plurality of subjectareas.

In accordance with the present invention, there is provided an imageprocessor for acquiring a processed frame by performing image processingon a desired frame sampled from a video image. The image processor ofthe present invention comprises:

similarity computation means for computing a similarity between thedesired frame and at least one frame which is temporally before andafter the desired frame; and

synthesis means for acquiring the processed frame by obtaining aweighting coefficient that becomes greater if the similarity becomesgreater, then weighting the at least one frame with the weightingcoefficient, and synthesizing the weighted frame and the desired frame.

In the image processor of the present invention, the aforementionedsimilarity computation means may partition the desired frame into aplurality of areas and compute the similarity for each of correspondingareas in at least one frame which correspond to the plurality of areas,and the aforementioned synthesis means may acquire the processed frameby obtaining weighting coefficients that become greater if thesimilarity becomes greater, then weighting the corresponding areas ofthe at least one frame with the weighting coefficients, and synthesizingthe weighted areas and the plurality of areas.

Also, in the image processor of the present invention, theaforementioned similarity computation means may partition the desiredframe into a plurality of subject areas that are included in the desiredframe and compute the similarity for each of corresponding subject areasin at least one frame which correspond to the plurality of subjectareas, and the aforementioned synthesis means may acquire the processedframe by obtaining weighting coefficients that become greater if thesimilarity becomes greater, then weighting the corresponding subjectareas of the at least one frame with the weighting coefficients, andsynthesizing the weighted subject areas and the plurality of subjectareas.

Note that the image processing method of the present invention may beprovided as a program to be executed by a computer .

There is a method of reducing image blurring by synthesizing a pluralityof images that have the same scene. Therefore, if a plurality of framesare sampled from a video image and synthesized, a synthesized frame canhave high picture quality. However, if a plurality of frames are merelysynthesized, the picture quality of the synthesized frame will bedegraded because a subject in a video image is in motion.

The image processing method and image processor of the present inventioncompute a similarity between a desired frame and at least one framewhich is temporally before and after the desired frame, and acquire aprocessed frame by obtaining a weighting coefficient that becomesgreater if the similarity becomes greater, then weighting the at leastone frame with the weighting coefficient, and synthesizing the weightedframe and the desired frame.

Therefore, there is no possibility that a dissimilar frame, as it is,will be added to a desired frame. This can reduce the influence ofdissimilar frames. Consequently, a processed frame with high picturequality can be obtained while reducing blurring that is caused bysynthesis of frames whose similarity is low.

According to the image processing method and image processor of thepresent invention, the desired frame is partitioned into a plurality ofareas. Also, the similarity is computed for each of corresponding areasin at least one frame which correspond to the plurality of areas. Theprocessed frame is acquired by obtaining weighting coefficients thatbecome greater if the similarity becomes greater, then weighting thecorresponding areas of the at least one frame with the weightingcoefficients, and synthesizing the weighted areas and the plurality ofareas. Therefore, even when a certain area in a video image is moved,blurring can be removed for each area. Thus, a processed frame withhigher picture quality can be obtained.

Also, the desired frame is partitioned into a plurality of subject areasthat are included in the desired frame. The similarity is computed foreach of corresponding subject areas in at least one frame whichcorrespond to the plurality of subject areas. The processed frame isacquired by obtaining weighting coefficients that become greater if thesimilarity becomes greater, then weighting the corresponding subjectareas of the at least one frame with the weighting coefficients, andsynthesizing the weighted subject areas and the plurality of subjectareas. Therefore, even when a certain subject area in a video image isin motion, blurring can be removed for each subject area. Thus, aprocessed frame with higher picture quality can be obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described in further detail with referenceto the accompanying drawings wherein:

FIG. 1 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a first embodiment of the presentinvention;

FIGS. 2A to 2D are diagrams for explaining the estimation of acorrespondent relationship between frames Fr_(N) and Fr_(N+1);

FIG. 3 is a diagram for explaining the deformation of patches;

FIG. 4 is a diagram for explaining a correspondent relationship betweena patch P1 and a reference patch P0;

FIG. 5 is a diagram for explaining bilinear interpolation;

FIG. 6 is a diagram for explaining the assignment of frame Fr_(N+1) to asynthesized image;

FIG. 7 is a diagram for explaining the computation of pixel values,represented by integer coordinates, in a synthesized image;

FIG. 8 is a diagram showing a graph for computing a weightingcoefficient;

FIG. 9 is a flowchart showing processes that are performed in the firstembodiment;

FIG. 10 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a second embodiment of the presentinvention;

FIG. 11 is a diagram showing an example of a low-pass filter;

FIG. 12 is a diagram showing a graph for computing a weightingcoefficient;

FIG. 13 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a third embodiment of the presentinvention;

FIG. 14 is a diagram showing a Laplacian filter;

FIG. 15 is a diagram showing a graph for computing a weightingcoefficient;

FIG. 16 is a flowchart showing processes that are performed in the thirdembodiment;

FIG. 17 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a fourth embodiment of the presentinvention;

FIG. 18 is a block diagram showing the construction of the samplingmeans of the video image synthesizer constructed in accordance with thefourth embodiment;

FIG. 19 is a diagram showing an example of a frame-number determinationtable;

FIG. 20 is a block diagram showing the construction of the stoppagemeans of the video image synthesizer constructed in accordance with thefourth embodiment;

FIG. 21 is a flowchart showing processes that are performed in thefourth embodiment;

FIG. 22 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a fifth embodiment of the presentinvention;

FIG. 23 is a block diagram showing the construction of the samplingmeans of the video image synthesizer constructed in accordance with thefifth embodiment;

FIG. 24 is a flowchart showing processes that are performed in the fifthembodiment;

FIG. 25 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a sixth embodiment of the presentinvention;

FIG. 26 is a block diagram showing the construction of the samplingmeans of the video image synthesizer constructed in accordance with thesixth embodiment;

FIGS. 27A and 27B are diagrams to explain the construction of firstextraction means in the sampling means shown in FIG. 26;

FIG. 28 is a diagram showing the construction of second extraction meansin the sampling means shown in FIG. 26;

FIG. 29 is a flowchart showing processes that are performed in the sixthembodiment;

FIG. 30 is a schematic block diagram showing a video image synthesizerconstructed in accordance with a seventh embodiment of the presentinvention;

FIG. 31 is a block diagram showing the construction of the samplingmeans of the video image synthesizer constructed in accordance with theseventh embodiment;

FIG. 32 is a schematic block diagram showing an image processorconstructed in accordance with an eighth embodiment of the presentinvention;

FIG. 33 is a diagram to explain the computation of similarities in theeighth embodiment;

FIGS. 34A and 34B are diagrams to explain the contributory degree of aframe to a pixel value;

FIG. 35 is a flowchart showing processes that are performed in theeighth embodiment;

FIG. 36 is a schematic block diagram showing an image processorconstructed in accordance with a ninth embodiment of the presentinvention;

FIG. 37 is a diagram to explain the computation of a similarity for eachregion;

FIG. 38 is a flowchart showing processes that are performed in the ninthembodiment;

FIG. 39 is a schematic block diagram showing an image processorconstructed in accordance with a tenth embodiment of the presentinvention;

FIG. 40 is a diagram to explain the computation of a motion vector foreach region;

FIGS. 41A and 41B are diagrams to explain how frame Fr₁ is partitionedinto a plurality of subject areas;

FIG. 42 is a diagram showing an example of a histogram; and

FIG. 43 is a flowchart showing processes that are performed in the tenthembodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will hereinafter be described indetail with reference to the drawings.

FIG. 1 shows a video image synthesizer constructed in accordance with afirst embodiment of the present invention. As illustrated in the figure,the video image synthesizer is equipped with sampling means 1 forsampling a plurality of frames from input video image data M0;correspondent relationship estimation means 2 for estimating acorrespondent relationship between a pixel in a reference frame and apixel in each frame other than the reference frame; coordinatetransformation means 3 for obtaining a coordinate-transformed frameFr_(T0) by transforming the coordinates of each frame (other than thereference frame) to the coordinate space of the reference frame on thebasis of the correspondent relationship estimated in the correspondentrelationship estimation means 2; and spatio-temporal interpolation means4 for obtaining a first interpolated frame Fr_(H1) whose resolution ishigher than each frame by interpolating each frame on the basis of thecorrespondent relationship estimated in the correspondent relationshipestimation means 2. The video image synthesizer is further equipped withspatial interpolation means 5 for obtaining a second interpolated frameFr_(H2) whose resolution is higher than each frame by interpolating thereference frame; correlation-value computation means 6 for computing acorrelation value that represents a correlation between thecoordinate-transformed frame Fr_(T0) and the reference frame;weighting-coefficient computation means 7 for computing a weightingcoefficient that is used in weighting the first interpolated frameFr_(H1) and the second interpolated frame Fr_(H2), on the basis of thecorrelation value computed in the correlation-value computation means 6;and synthesis means 8 for acquiring a synthesized frame Fr_(G) byweighting the first interpolated frame Fr_(H1) and the secondinterpolated frame Fr_(H2) on the basis of the weighting coefficientcomputed by the weighting-coefficient computation means 7. In the firstembodiment, it is assumed that the number of pixels in the longitudinaldirection of the synthesized frame Fr_(G) and the number of pixels inthe transverse direction are twice those of a sampled frame,respectively. In the following description, while the numbers of pixelsin the longitudinal and transverse directions of the synthesized frameFr_(G) are respectively double those of a sampled frame, they may be ntimes (where n is a positive number), respectively.

The sampling means 1 is used to sample a plurality of frames from videoimage data M0, but in the first embodiment two frames Fr_(N) andFr_(N+1) are sampled from the video image data M0. It is assumed thatthe frame Fr_(N) is a reference frame. The video image data M0represents a color video image, and each of the frames Fr_(N) andFr_(N+1) consists of a luminance (monochrome brightness) component (Y)and two color difference components (Cb and Cr). In the followingdescription, processes are performed on the three components, but arethe same for each component. Therefore, in the first embodiment, adetailed description will be given of processes that are performed onthe luminance component Y, and a description of processes that areperformed on the color difference components Cb and Cr will not be made.

The correspondent relationship estimation means 2 estimates acorrespondent relationship between the reference frame Fr_(N) and thesucceeding frame Fr_(N+1) in the following manner. FIGS. 2A to 2D arediagrams for explaining the estimation of a correspondent relationshipbetween the reference frame Fr_(N) and the succeeding frame Fr_(N+1). Itis assumed that in the figures, a circular subject within the referenceframe Fr_(N) has been slightly moved rightward in the succeeding frameFr_(N+1).

First, the correspondent relationship estimation means 2 places areference patch P0 consisting of one or a plurality of rectangular areason the reference frame Fr_(N). FIG. 2A shows the state in which thereference patch P0 is placed on the reference frame Fr_(N). Asillustrated in the figure, in the first embodiment, the reference patchP0 consists of sixteen rectangular areas, arranged in a 4×4 format.Next, as illustrated in FIG. 2B, the same patch P1 as the referencepatch P0 is placed at a suitable position on the succeeding frameFr_(N+1), and a correlation value, which represents a correlationbetween an image within the reference patch P0 and an image within thepatch P1, is computed. Note that the correlation value can be computedas a mean square error by the following Formula 1. As shown in FIG. 2A,the x axis extends along the horizontal axis and the y axis extendsalong the vertical direction.

$\begin{matrix}{E = {\frac{1}{N}{\sum\limits_{i}^{N}\; \left( {{pi} - {qi}} \right)^{2}}}} & (1)\end{matrix}$

in which

E=correlation value,

pi and qi=pixel values of corresponding pixels within the referencepatch P0 and the patch P1,

N=number of pixels within the reference patch P0 and the patch P1.

Next, the patch P1 on the succeeding frame Fr_(N+1) is moved in the fourdirections (up, down, right, and left directions) by constant pixelquantities ±Δx and ±Δy, and then a correlation value between an imagewithin the patch P1 and an image within the reference patch P0 withinthe reference frame Fr_(N) is computed. Correlation values arerespectively computed in the up, down, right, and left directions andobtained as E(Δx, 0), E(−Δx, 0), E(0, Δy), and E(0, −Δy).

From the four correlation values E(Δx, 0), E(−Δx, 0), E (0, Δy), and E(0, −Δy) after movement, a gradient direction in which a correlationvalue becomes smaller (i.e., a gradient direction in which a correlationbecomes greater) is obtained as a correlation gradient, and as shown inFIG. 2C, the patch P1 is moved in that direction by a predeterminedquantity equal to m times (where m is a real number). More specifically,coefficients C(Δx, 0), C(−Δx, 0), C(0, Δy), and C(0, −Δy) are computedby the following Formula 2, and from these coefficients, correlationgradients g_(x) and g_(y) are computed by the following Formulas 3 and4.

$\begin{matrix}{{c\left( {{\Delta \; x},{\Delta \; y}} \right)} = {\sqrt{E\left( {{\Delta \; x},{\Delta \; y}} \right)}/255}} & (2) \\{{gx} = \frac{{c\left( {{\Delta \; x},0} \right)} - {c\left( {{{- \Delta}\; x},0} \right)}}{2}} & (3) \\{{gy} = \frac{c\left( {\left( {0,{\Delta \; y}} \right) - {c\left( {0,{{- \Delta}\; y}} \right)}} \right.}{2}} & (4)\end{matrix}$

Based on the computed correlation gradients g_(x) and g_(y), the patchP1 is moved by (−λ1 g _(x), −λ1 g _(y)), and by repeating theaforementioned processes, the patch P1 is iteratively moved until itconverges at a certain position, as shown in FIG. 2D. The parameter λ1is used to determine the speed of convergence and is represented by areal number. If the value of λ1 is too great, then a solution willdiverge due to the iteration process and therefore it is necessary tochoose a suitable value (e.g., 10).

Further, a lattice point in the patch P1 is moved in the 4 directionsalong the coordinate axes by constant pixel quantities. When thisoccurs, a rectangular area containing the moved lattice point isdeformed as shown in FIG. 3, for example. Correlation values between thedeformed rectangular area and the corresponding rectangular area of thereference patch P0 are computed. These correlation values are assumed tobe E1(Δx, 0), E1(−Δx, 0), E1(0, Δy), and E1(0, −Δy).

As with the aforementioned case, from the 4 correlation values E1(Δx,0), E1(−Δx, 0), E1(0, Δy), and E1(0, −Δ_(y)) after deformation, agradient direction in which a correlation value becomes smaller (i.e., agradient direction in which a correlation becomes greater) is obtainedas a correlation gradient, and a lattice point in the patch P1 is movedin that direction by a predetermined quantity equal to m times (where mis a real number). This is performed on all the lattice points of thepatch P1 and referred to as a single processing. This processing isrepeatedly performed until the coordinates of the lattice pointsconverge.

In this manner, the moved quantity and deformed quantity of the patch P1with respect to the reference patch P0 are computed, and based on thesequantities, a correspondent relationship between a pixel within thereference patch P0 of the reference frame Fr_(N) and a pixel within thepatch P1 of the succeeding frame Fr_(N+1) can be estimated.

The coordinate transformation means 3 transforms the coordinates of thesucceeding frame Fr_(N+1) to the coordinate space of the reference frameFr_(N) and obtains a coordinate-transformed frame Fr_(T0), as describedbelow. In the following description, transformation, interpolation, andsynthesis are performed only on the areas within the reference patch P0of the reference frame Fr_(N) and areas within the patch P1 of thesucceeding frame Fr_(N+1).

In the first embodiment, the coordinate transformation is performedemploying bilinear transformation. The coordinate transformation bybilinear transformation is defined by the following Formulas 5 and 6.

x=(1−u)(1−v)x1+(1−v)ux2+(1−u)vx3+uvx4   (5)

y=(1−u)(1−v)y1+(1−v)uy2+(1−u)vy3+uvy4   (6)

Using Formulas 5 and 6, the coordinates within the patch P1 representedby 4 points (xn, yn) (1≦n≦4) at two-dimensional coordinates areinterpolated by a normalized coordinate system (u, v) (0≦u, v≦1). Thecoordinate transformation within two arbitrary rectangles can beperformed by combining Formulas 5 and 6 and inverse transformation ofFormulas 5 and 6.

Now, consider how a point (x, y) within the patch P1 (xn, yn)corresponds to a point (x′, y′) within the reference patch P0 (x′n,y′n), as illustrated in FIG. 4. First, a point (x, y) within the patchP1 (xn, yn) is transformed to normalized coordinates (u, v), which arecomputed by inverse transformation of Formulas 5 and 6. Based on thereference patch P0 (x′n, y′n) corresponding to the normalizedcoordinates (u, v), coordinates (x′, y′) corresponding to the point (x,y) are computed by Formulas 5 and 6. The coordinates of a point (x, y)are integer coordinates where pixel values are originally present, butthere are cases where the coordinates of a point (x′, y′) become realcoordinates where no pixel value is present. Therefore, pixel values atinteger coordinates after transformation are computed as the sum of theweighted pixel values of coordinates (x′, y′), transformed within anarea that is surrounded by 8 neighboring integer coordinates adjacent tointeger coordinates in the reference patch P0.

More specifically, integer coordinates b (x, y) in the reference patchP0, as shown in FIG. 5, are computed based on pixel values in thesucceeding frame Fr_(N+1), transformed within an area that is surroundedby the 8 neighboring integer coordinates b(x−1, y−1), b(x, y−1), b(x+1,y−1), b(x−1, y), b(x+1, y), b(x−1, y+1), b(x, y+1), and b(x+1, y+1). Ifm pixel values in the succeeding frame Fr_(N+1) are transformed withinan area that is surrounded by 8 neighboring pixels, and the pixel valueof each pixel transformed is represented by I_(tj) (x^(∘),y^(∘))(1≦j≦m), then a pixel value I_(t)(x^(̂), y^(̂)) at integercoordinates b (x, y) can be computed by the following Formula 7. Notethat φ in Formula 7 is a function representing the sum of weightedvalues.

$\begin{matrix}\begin{matrix}{{I_{t}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = {\varphi \left( {I_{tj}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} \right)}} \\{= {\left\{ \begin{pmatrix}{{W_{1} \times {I_{t\; 1}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} + \left( {{W_{2} \times {I_{t\; 2}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} + \ldots \mspace{14mu} +} \right.} \\\left( {W_{m} \times {I_{t\; m}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} \right.\end{pmatrix} \right\}/}} \\{\left( {W_{1} + W_{2} + \ldots + W_{k}} \right)} \\{= \frac{\sum\limits_{j = 1}^{m}\; {W_{i} \times {I_{tj}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}}{\sum\limits_{j = 1}^{m}\; W_{i}}}\end{matrix} & (7)\end{matrix}$

in which

Wi (1≦j≦m)=product of coordinate interior division ratios viewed fromneighboring integer pixels at a position where a pixel valueI_(tj)(x^(∘), y^(∘)) is assigned.

For simplicity, consider the case where two pixel values I_(t1) andI_(t2) in the succeeding frame Fr_(N+1) are transformed within an areasurrounded by 8 neighboring pixels, employing FIG. 5. A pixel valueI_(t)(x^(̂), y^(̂)) at integer coordinates b (x, y) can be computed by thefollowing Formula 8.

I _(t)(x ^(̂) , y ^(̂))=(W1×W2)=W1×I _(t1) +W2×I _(t2))   (8)

in which

-   -   W1=u×v, and    -   W2=(1−s)×(1−t).

By performing the aforementioned processing on all pixels within thepatch P1, an image within the patch P1 is transformed to a coordinatespace in the reference frame Fr_(N), whereby a coordinate-transformedframe Fr_(T0) is obtained.

The spatio-temporal interpolation means 4 interpolates the succeedingframe Fr_(N+1) and obtains a first interpolated frame Fr_(H1). Morespecifically, a synthesized image with the finally required number ofpixels is first prepared as shown in FIG. 6. (In the first embodiment,the numbers of pixels in the longitudinal and transverse directions of asynthesized image are respectively double those of the sampled frameFr_(N) or Fr_(N+1), but they may be n times the number of pixels(wherein n is a positive number), respectively.) Then, based on thecorrespondent relationship obtained by the correspondent relationshipestimation means 2, the pixel values of pixels in the succeeding frameFr_(N+1) (areas within the patch P1) are assigned to the synthesizedimage. If a function for performing this assignment is represented by H,the pixel value of each pixel in the succeeding frame Fr_(N+1) isassigned to the synthesized image by the following Formula 9.

I _(1N+1)(x ^(∘) , y ^(∘))=H(Fr _(N+1)(x, y))   (9)

in which

-   -   I_(1N+1)(x^(∘), y^(∘))=pixel value in the succeeding frame        Fr_(N+1), assigned to the synthesized image,    -   Fr_(N+1)(x, y)=pixel value in the succeeding frame Fr_(N+1).

Thus, by assigning the pixel values in the succeeding frame Fr_(N+1) tothe synthesized image, a pixel value I_(1N+1)(x^(∘), y^(∘)) is obtainedand the first interpolated frame Fr_(H1) with a pixel value I₁(x^(∘),y^(∘))(=I_(1N+1)(x^(∘), y^(∘))) for each pixel is obtained.

In assigning pixel values to a synthesized image, there are cases whereeach pixel in the succeeding frame Fr_(N+1) does not correspond to theinteger coordinates (i.e., coordinates in which pixel values should bepresent) of the synthesized image, depending on the relationship betweenthe number of pixels in the synthesized image and the number of pixelsin the succeeding frame Fr_(N+1). In the first embodiment, pixel valuesat the integer coordinates of a synthesized image are computed at thetime of synthesis, as described later. But, to make a description at thetime of synthesis easier, the computation of pixel values at the integercoordinates of a synthesized image will hereinafter be described.

The pixel values at the integer coordinates of a synthesized image arecomputed as the sum of the weighted pixel values of pixels in thesucceeding frame Fr_(N+1), assigned within an area that is surrounded by8 neighboring integer coordinates adjacent to the integer coordinates ofthe synthesized image.

More specifically, integer coordinates p(x, y) in a synthesized image,as shown in FIG. 7, are computed based on pixel values in the succeedingframe Fr_(N+1), assigned within an area that is surrounded by the 8neighboring integer coordinates p(x−1, y−1), p(x, y−1), p(x+1, y−1),p(x−1, y), p(x+1, y), p(x−1, y+1), p(x, y+1), and p(x+1, y+1). If kpixel values in the succeeding frame Fr_(N+1) are assigned within anarea that is surrounded by 8 neighboring pixels, and the pixel value ofeach pixel assigned is represented by I_(1N+1i)(x^(∘), y^(∘)) (1≦i≦k),then a pixel value I_(1N+1)(x^(̂), y^(̂)) at integer coordinates p (x, y)can be computed by the following Formula 10. Note that φ in Formula 10is a function representing the sum of weighted values.

$\begin{matrix}\begin{matrix}{{I_{{1N} + 1}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = {\varphi \left( {I_{{1N} + 1}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} \right)}} \\{= {\left\{ \begin{pmatrix}{{M_{1} \times {I_{{1N} + 11}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} +} \\{{M_{2} \times {I_{{1N} + 12}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} + \ldots +} \\{{Mk} \times {I_{{1N} + {1k}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}\end{pmatrix} \right\}/}} \\{\left( {W_{1} + W_{2} + \ldots + W_{k}} \right)} \\{= \frac{\sum\limits_{i = 1}^{k}\; {M_{i} \times I_{{1N} + 1}{{iM}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}}{\sum\limits_{j = 1}^{m}\; M_{i}}}\end{matrix} & (10)\end{matrix}$

in which

-   -   Mi (1≦i≦k)=product of coordinate interior division ratios viewed        from neighboring integer pixels at a position where a pixel        value I_(1N+1i)(x^(∘), y^(∘)) is assigned.

For simplicity, consider the case where two pixel values I_(1N+11) andI_(1N+12) in the succeeding frame Fr_(N+1) are assigned within an areasurrounded by 8 neighboring pixels, employing FIG. 7. A pixel valueI_(1N+1)(x^(̂), y^(̂)) at integer coordinates p (x, y) can be computed bythe following Formula 11.

I _(1N+1)(x ^(̂) , y ^(̂))=(M1+M2)=(M1×I _(1N+11) +M2×I _(1N+12))   (11)

in which

-   -   M1=u×v, and    -   M2=(1−s)×(1−t).

By assigning a pixel value in the succeeding frame FrN+1 to all integercoordinates of a synthesized image, a pixel value I1N+1 (x^(̂), y^(̂)) canbe obtained. In this case, each pixel value I1(x^(̂), y^(̂)) in the firstinterpolated frame FrH1 becomes I1N+1 (x^(̂), y^(̂)).

While the first interpolated frame Fr_(H1) is obtained by interpolatingthe succeeding frame Fr_(N+1), the first interpolated frame Fr_(H1) maybe obtained employing the reference frame Fr_(N) as well as thesucceeding frame Fr_(N+1). In this case, pixels in the reference frameFr_(N) are interpolated and directly assigned to integer coordinates ofa synthesized image.

The spatial interpolation means 5 obtains a second interpolated frameFr_(H2) by performing interpolation, in which pixel values are assignedto coordinates (real coordinates (x^(∘), y^(∘)) to which pixels in thesucceeding frame Fr_(N+1) on a synthesized image are assigned, on thereference frame Fr_(N). Assuming a pixel value at the real coordinatesof the second interpolated frame Fr_(H2) is I₂(x^(∘), y^(∘)), the pixelvalue I₂(x^(∘), y^(∘)) is computed by the following Formula 12.

I ₂(x ^(∘) , y ^(∘))=f(Fr _(N)(x, y))   (12)

where f is an interpolation function.

Note that the aforementioned interpolation can employ linearinterpolation, spline interpolation, etc.

In the first embodiment, the numbers of pixels in longitudinal andtransverse directions of a synthesized frame are two times those of thereference frame Fr_(N), respectively. Therefore, by interpolating thereference frame Fr_(N) so that the numbers of pixels in the longitudinaland transverse directions double, a second interpolated frame Fr_(H2)with a number of pixels corresponding to the number of pixels of asynthesized image may be obtained. In this case, a pixel value to beobtained by interpolation is a pixel value at integer coordinates in asynthesized image, so if this pixel value is I₂(x^(̂), y^(̂)), the pixelvalue I₂(x^(̂), y^(̂)) is computed by the following Formula 13.

I ₂(x ^(̂) , y ^(̂))=f(Fr _(N)(x, y))   (13)

The correlation-value computation means 6 computes a correlation valued0(x, y) between corresponding pixels of a coordinate-transformed frameFr_(T0) and reference frame Fr_(N). More specifically, as indicated inthe following Formula 14, the absolute value of a difference between thepixel values Fr_(T0) (x, y) and Fr_(N)(x, y) of corresponding pixels ofthe coordinate-transformed frame Fr_(T0) and reference frame Fr_(N) iscomputed as the correlation value d0(x, y). Note that the correlationvalue d0(x, y) becomes a smaller value if the correlation between thecoordinate-transformed frame Fr_(T0) and the reference frame Fr_(N)becomes greater.

d0(x, y)=|Fr _(T0)(x, y)−Fr _(N)(x, y)|  (14)

In the first embodiment, the absolute value of a difference between thepixel values Fr_(T0) (x, y) and Fr_(N)(x, y) of corresponding pixels ofthe coordinate-transformed frame Fr_(T0) and reference frame Fr_(N) iscomputed as the correlation value d0(x, y). Alternatively, the square ofthe difference maybe computed as the correlation value. Also, while thecorrelation value is computed for each pixel, it may be obtained foreach area by partitioning the coordinate-transformed frame Fr_(T0) andreference frame Fr_(N) into a plurality of areas and then computing theaverage or sum of all pixel values within each area. In addition, bycomputing the average or sum of the correlation values d0(x, y) computedfor the entire frame, the correlation value may be obtained for eachframe. Further, by respectively computing histograms for thecoordinate-transformed frame Fr_(T0) and the reference frame Fr_(N), theaverage value, median value, or standard-deviation difference value ofthe histograms for the coordinate-transformed frame Fr_(T0) andreference frame Fr_(N), or the accumulation of histogram differencevalues, maybe employed as the correlation value. Moreover, by computingfor each pixel or each small area a motion vector that represents themotion of the coordinate-transformed frame Fr_(T0) with respect to thereference frame Fr_(N), the average value, median value, or standarddeviation of computed motion vectors may be employed as the correlationvalue, or the histogram accumulation of motion vectors may be employedas the correlation value.

The weighting-coefficient computation means 7 acquires a weightingcoefficient α(x, y) that is used in weighting the first interpolatedframe Fr_(H1) and second interpolated frame Fr_(H2), from thecorrelation value d0(x, y) computed by the correlation-value computationmeans 6. More specifically, the weighting-coefficient computation means7 acquires a weighting coefficient α(x, y) by referring to a graph shownin FIG. 8. As illustrated in the figure, if the correlation value d0(x,y) becomes smaller, that is, if the correlation between thecoordinate-transformed frame Fr_(T0) and the reference frame Fr_(N)becomes greater, the value of the weighting coefficient α(x, y) becomescloser to zero. Note that the correlation value d0(x, y) is representedby a 8-bit value.

Further, the weighting-coefficient computation means 7 computes aweighting coefficient α(x^(∘), y^(∘)) at coordinates (real coordinates)to which pixels in the succeeding frame Fr_(N+1) are assigned, byassigning the weighting coefficient α(x, y) to a synthesized image, asin the case where pixels in the succeeding frame Fr_(N+1) are assignedto a synthesized image. More specifically, as with the interpolationperformed by the spatial interpolation means 5, the weightingcoefficient α(x^(∘), y^(∘)) is acquired by performing interpolation, inwhich pixel values are assigned to coordinates (real coordinates (x^(∘),y^(∘))) to which pixels in the succeeding frame Fr_(N+1) on asynthesized image are assigned, on the weighting coefficient α(x, y).

By enlarging or equally multiplying the reference frame Fr_(N) so thatit becomes equal to the size of a synthesized image to acquire anenlarged or equally-multiplied reference frame, without computing theweighting coefficient α(x^(∘), y^(∘)) at the real coordinates in asynthesized image by interpolation, a weighting coefficient α(x, y),acquired for a pixel of the enlarged or equally-multiplied referenceframe that is closest to real coordinates to which the pixels of thesucceeding frame Fr_(N+1) in the synthesized image are assigned, may beemployed as the weighting coefficient α(x^(∘), y^(∘)) at the realcoordinates.

Further, in the case where pixel values I₁(x^(̂), y^(̂)) and I₂(x^(̂),y^(̂)) at integer coordinates in a synthesized image have been acquired,a weighting coefficient α(x^(̂), y^(̂)) at the integer coordinates in thesynthesized image may be computed by computing the sum of the weightedvalues of the weighting coefficients α(x^(∘), y^(∘)) assigned to thesynthesized image in the aforementioned manner.

The synthesis means 8 weights and adds the first interpolated frameFr_(H1) and the second interpolated frame Fr_(H2) on the basis of theweighting coefficient α(x^(∘), y^(∘)) computed by theweighting-coefficient computation means 7, thereby acquiring asynthesized frame Fr_(G) that has a pixel value Fr_(G) (x^(̂), y^(̂)) atthe integer coordinates of a synthesized image. More specifically, thesynthesis means 8 weights the pixel values I₁(x^(∘), y^(∘)) andI₂(x^(∘), y^(∘)) of corresponding pixels of the first interpolated frameFr_(H1) and second interpolated frame Fr_(H2) on the basis of theweighting coefficient α(x^(∘), y^(∘)) and also adds the weighted values,employing the following Formula 15. In this manner, the pixel valueFr_(G)(x^(̂), y^(̂)) of a synthesized frame Fr_(G) is acquired.

$\begin{matrix}{{{Fr}_{G}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = \frac{\sum\limits_{i = 1}^{k}\; {M_{i} \times \begin{bmatrix}{{I\; 2{i\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} + {\alpha \; {i\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} \times}} \\\left\{ {{I\; 1{i\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} - {I\; 2{i\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}} \right\}\end{bmatrix}}}{\sum\limits_{i = 1}^{k}\; M_{i}}} & (15)\end{matrix}$

In Formula 15, k is the number of pixels in the succeeding frameFr_(N+1) assigned to an area that is surrounded by 8 neighboring integercoordinates of integer coordinates (x^(̂), y^(̂)) of a synthesized frameFr_(G) (i.e., a synthesized image), and these assigned pixels have pixelvalues I₁(x^(∘), y^(∘)) and I₂(x^(∘), y^(∘)) and weighting coefficientα(x^(∘), y^(∘)).

In the first embodiment, if the correlation between the reference frameFr_(N) and the coordinate-transformed frame Fr_(T0) becomes greater, theweight of the first interpolated frame Fr_(H1) is made greater. In thismanner, the first interpolated frame Fr_(H1) and second interpolatedframe Fr_(H2) are weighted and added.

Note that there are cases where pixel values cannot be assigned to allinteger coordinates of a synthesized image. In such a case, pixel valuesat integer coordinates not assigned can be computed by performinginterpolation on assigned pixels in the same manner as the spatialinterpolation means 5.

While the process of acquiring the synthesized frame FrG for theluminance component Y has been described, synthesized frames FrG forcolor difference components Cb and Cr are acquired in the same manner.By combining a synthesized frame FrG(Y) obtained from the luminancecomponent Y and synthesized frames FrG(Cb) and FrG(Cr) obtained from thecolor difference components Cb and Cr, a final synthesized frame isobtained. To expedite processing, it is preferable to estimate acorrespondent relationship between the reference frame FrN and thesucceeding frame FrN+1 only for the luminance component Y, and processthe color difference components Cb and Cr on the basis of thecorrespondent relationship estimated for the luminance component Y.

In the case where the first interpolated frame Fr_(H1) and secondinterpolated frame Fr_(H2) having pixel values for the integercoordinates of a synthesized image, and the weighting coefficientα(x^(̂), y^(̂)) at the integer coordinates, have been acquired, a pixelvalue Fr_(G)(x, y) in the synthesized frame Fr_(G) can be acquired byweighting and adding the pixel values I₁(x^(̂), y^(̂)) and I₂(x^(̂), y^(̂))of corresponding pixels of the first interpolated frame Fr_(H1) andsecond interpolated frame Fr_(H2) on the basis of the weightingcoefficient α(x^(̂), y^(̂)), employing the following Formula 16.

Fr _(G)(x ^(̂) ,y ^(̂))=α(x ^(̂) , y ^(̂))×I ₁(x ^(̂) , y ^(̂))+{1−α(x ^(̂) , y^(̂))}×I ₂(x ^(̂) , y ^(̂))   (16)

Now, a description will be given of operation of the first embodiment.FIG. 9 shows processes that are performed in the first embodiment. Inthe following description, the first interpolated frame Fr_(H1), secondinterpolated frame Fr_(H2), and weighting coefficient α(x^(∘), y^(∘))are obtained at real coordinates to which pixels in the frame Fr_(H1+1)of a synthesized image are assigned. First, video image data M0 is inputto the sampling means 1 (step S1). In the sampling means 1, a referenceframe Fr_(N) and the succeeding frame Fr_(N+1) are sampled from theinput video image data M0 (step S2). Then, a correspondent relationshipbetween the reference frame Fr_(N) and the succeeding frame Fr_(N+1) isestimated by the correspondent relationship estimation means 2 (stepS3).

Based on the correspondent relationship estimated by the correspondentrelationship estimation means 2, the coordinates of the succeeding frameFrN+1 are transformed to the coordinate space in the reference frame FrNby the coordinate transformation means 3, whereby acoordinate-transformed frame FrT0 is acquired (step S4). The correlationvalue d0(x, y) of corresponding pixels of the coordinate-transformedframe FrT0 and reference frame FrN is computed by the correlation-valuecomputation means 6 (step S5). Further, the weight computation means 7computes a weighting coefficient α(x^(∘), y^(∘)), based on thecorrelation value d0(x, y) (step S6).

On the other hand, based on the correspondent relationship estimated bythe correspondent relationship estimation means 2, a first interpolatedframe Fr_(H1) is acquired by the spatio-temporal interpolation means 4(step S7), and a second interpolated frame Fr_(H2) is acquired by thespatial interpolation means 5 (step S8).

Note that the processes in steps S7 and S8 may be previously performedand the processes in steps S4 to S6 and the processes in steps S7 and S8may be performed in parallel.

In the synthesis means 8, a pixel value I1(x^(∘), y^(∘)) in the firstinterpolated frame FrH1 and a pixel value I2(x^(∘), y^(∘)) in the secondinterpolated frame FrH2 are synthesized, whereby a synthesized frame FrGconsisting of a pixel value FrG(x^(̂), y^(̂)) is acquired (step S9), andthe processing ends.

In the case where the motion of subjects included in the reference frameFr_(N) and succeeding frame Fr_(N+1) is small, the first interpolatedframe Fr_(H1) represents a high-definition image whose resolution ishigher than the reference frame Fr_(N) and succeeding frame Fr_(N+1). Onthe other hand, in the case where the motion of subjects included in thereference frame Fr_(N) and succeeding frame Fr_(N+1) is great orcomplicated, a moving subject in the first interpolated frame Fr_(H1)becomes blurred.

In addition, the second interpolated frame Fr_(H2) is obtained byinterpolating only one reference frame Fr_(N), so it is inferior indefinition to the first interpolated frame Fr_(H1), but even when themotion of a subject is great or complicated, the second interpolatedframe Fr_(H2) does not blur so badly because it is obtained from onlyone reference frame Fr_(N).

Furthermore, the weighting coefficient α(x^(∘), y^(∘)) to be computed bythe weight computation means 7 is set so that if the correlation betweenthe reference frame Fr_(N) and the coordinate-transformed frame Fr_(T0)becomes greater, the weight of the first interpolated frame Fr_(H1)becomes greater.

If the motion of a subject included in each of the frames Fr_(N) andFr_(N+1) is small, the correlation between the coordinate-transformedframe Fr_(T0) and the reference frame Fr_(N) becomes great, but if themotion is great or complicated, the correlation becomes small.Therefore, by weighting the first interpolated frame Fr_(H1) and secondinterpolated frame Fr_(H2) on the basis of the weighting coefficientα(x^(∘), y^(∘)) computed by the weight computation means 7, when themotion of a subject is small there is obtained a synthesized frameFr_(G) in which the ratio of the first interpolated frame Fr_(H1) withhigh definition is high, and when the motion is great there is obtaineda synthesized frame Fr_(G) including at a high ratio the secondinterpolated frame Fr_(H2) in which the blurring of a moving subject hasbeen reduced.

Therefore, in the case where the motion of a subject included in each ofthe frames Fr_(N) and Fr_(N+1) is great, the blurring of a subject inthe synthesized frame Fr_(G) is reduced, and when the motion is small,high definition is obtained. In this manner, a synthesized frame Fr_(G)with high picture quality can be obtained regardless of the motion of asubject included in each of the frames Fr_(N) and Fr_(N+1).

Now, a description will be given of a second embodiment of the presentinvention. FIG. 10 shows a video image synthesizer constructed inaccordance with the second embodiment of the present invention. Becausethe same reference numerals will be applied to the same parts as thefirst embodiment, a detailed description of the same parts will not begiven.

The second embodiment differs from the first embodiment in that it isprovided with filter means 9. The filter means 9 performs a filteringprocess on a correlation value d0(x, y) computed by correlation-valuecomputation means 6, employing a low-pass filter.

An example of the low-pass filter is shown in FIG. 11. The secondembodiment employs a 3×3 low-pass filter, but may employ a 5×5 low-passfilter or greater. Alternatively, a median filter, a maximum valuefilter, or a minimum value filter may be employed.

In the second embodiment, with weight computation means 7 a weightingcoefficient α(x^(∘), y^(∘)) is acquired based on the correlation valued0′(x, y) filtered by the filter means 9, and the weighting coefficientα(x^(∘), y^(∘)) is employed in the weighting and addition operationsthat are performed in the synthesis means 8.

Thus, in the second embodiment, a filtering process is performed on thecorrelation value d0(x, y) through a low-pass filter, and based on thecorrelation value d0′(x, y) obtained in the filtering process, theweighting coefficient α(x^(∘), y^(∘)) is acquired. Because of this, achange in the weighting coefficient α(x^(∘), y^(∘)) in the synthesizedimage becomes smooth, and consequently, image changes in areas wherecorrelation values change can be smoothed. This is able to give thesynthesized frame Fr_(G) a natural look.

In the above-described first and second embodiments and the followingembodiments, while the correlation value d0(x, y) is computed for theluminance component Y and color difference components Cb and Cr, aweighting coefficient α(x, y) may be computed for the luminancecomponent Y and color difference components Cb and Cr by weighting andadding a correlation value d0Y(x, y) for the luminance component andcorrelation values d0Cb(x, y) and d0Cr(x, y) for the color differencecomponents, employing weighting coefficients a, b, and c, as shown inthe following Formula 17.

d1(x, y)=a·d0Y(x, y)+b·d0Cb(x, y)+c·d0Cr(x, y)   (17)

By computing a Euclidean distance employing the luminance componentFr_(T0Y)(x, y) and color difference components Fr_(T0Cb)(x, y) andFr_(T0Cr)(x, y) of the coordinate-transformed frame Fr_(T0), theluminance component Fr_(NY)(x, y) and color difference componentsFr_(NCb)(x, y) and Fr_(NCr)(x, y) of the reference frame Fr_(N), andweighting coefficients a, b, and c, as shown in the following Formula18, the computed Euclidean distance may be used as a correlation valued1(x, y) for acquiring a weighting coefficient α(x, y).

d1(x, y)={a(Fr _(T0Y)(x, y)−Fr _(NY)(x, y))² +b(Fr _(T0Cb)(x, y)−Fr_(NCb)(x, y))² +c(Fr _(T0Cr)(x, y)−Fr _(NCc)(x, y))²}^(0.5)   (18)

In the above-described first and second embodiments and the followingembodiments, although the weight computation means 7 acquires theweighting coefficient α(x, y) employing a graph shown in FIG. 8, theweight computation means 7 may employ a nonlinear graph in which thevalue of the weighting coefficient α(x, y) changes smoothly and slowlyat boundary portions where a value changes, as shown in FIG. 12.

Thus, by employing a nonlinear graph shown in FIG. 12, the degree of achange in an image becomes slow at local areas where correlation valueschange. This is able to give a synthesized frame a natural look.

In the above-described first and second embodiments and the followingembodiments, although a synthesized frame Fr_(G) is acquired from twoframes Fr_(N) and Fr_(N+1), it may be acquired from three or moreframes. For instance, in the case of acquiring a synthesized frameFr_(G) from T frames Fr_(N+t′) (0≦t′≦T−1), a correspondent relationshipbetween the reference frame Fr_(N)(=Fr_(N+0)) and each of the framesFr_(N+t) (0≦t≦T−1) other than the reference frame is estimated and aplurality of first interpolated frames Fr_(H1t) are obtained. Note thata pixel value in the first interpolated frame Fr_(H1t) is represented byI_(1t)(x^(∘), y^(∘)).

In addition, interpolation, in which pixel values are assigned tocoordinates (real coordinates (x^(∘), y^(∘))) where pixels of the frameFr_(N+t) in a synthesized image are assigned, is performed on thereference frame Fr_(N), whereby a second interpolated frame Fr_(H2t)corresponding to the frame Fr_(N+t) is acquired. Note that a pixel valuein the second interpolated frame Fr_(H2t) is represented byI_(2t)(x^(∘), y^(∘)).

Moreover, based on the correspondent relationship estimated, a weightingcoefficient αt(x^(∘), y^(∘)), for weighting first and secondinterpolated frames Fr_(H1t) and Fr_(H2t) that correspond to each other,is acquired.

By performing a weighting operation on corresponding first and secondinterpolated frames Fr_(H1t) and Fr_(H2t) by the weighting coefficientαt(x^(∘), y^(∘)) and also adding the weighted frames, an intermediatesynthesized frame Fr_(Gt) with a pixel value Fr_(Gt)(x^(̂), y^(̂)) atinteger coordinates in a synthesized image is acquired. Morespecifically, as shown in the following Formula 19, the pixel valuesI_(1t)(x^(∘), y^(∘)) and I_(2t)(x^(∘), y^(∘)) of corresponding pixels ofthe first and second interpolated frames Fr_(H1t) and Fr_(H2t) areweighted by employing the corresponding weighting coefficient αt(x^(∘),y^(∘)), and the weighted values are added. In this manner, the pixelvalue Fr_(Gt)(x^(̂), y^(̂)) of an intermediate synthesized frame Fr_(Gt)is acquired.

$\begin{matrix}{{{Fr}_{Gt}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = \frac{\sum\limits_{i = 1}^{k}\; {M_{ti} \times \begin{bmatrix}{{I_{2{ti}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} + {{\alpha_{ti}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} \times}} \\\left\{ {{{I\;}_{1{ti}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} - {{I\;}_{2{ti}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} \right\}\end{bmatrix}}}{\sum\limits_{i = 1}^{k}\; M_{ti}}} & (19)\end{matrix}$

In Formula 19, k is the number of pixels in the frame Fr_(N+t) assignedto an area that is surrounded by 8 neighboring integer coordinates inthe integer coordinates (x^(̂), y^(̂)) of an intermediate synthesizedframe Fr_(Gt) (i.e., a synthesized image), and these assigned pixelshave pixel values I_(1t)(x^(∘), y^(∘)) and I_(2t)(x^(∘), y^(∘)) andweighting coefficient αt(x^(∘), y^(∘)).

By adding the intermediate synthesized frames Fr_(Gt), a synthesizedframe Fr_(G) is acquired. More specifically, by adding correspondingpixels of intermediate synthesized frames Fr_(Gt) with the followingFormula 20, a pixel value Fr_(G) (x^(̂), y^(̂)) in a synthesized frameFr_(G) is acquired.

$\begin{matrix}{{{Fr}_{G}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = {\sum\limits_{t = 1}^{T - 1}\; {{Fr}_{Gt}\left( {x^{\bigwedge},y^{\bigwedge}} \right)}}} & (20)\end{matrix}$

Note that there are cases where pixel values cannot be assigned to allinteger coordinates of a synthesized image. In such a case, pixel valuesat integer coordinates not assigned can be computed by performinginterpolation on assigned pixels in the same manner as the spatialinterpolation means 5.

In the case of acquiring a synthesized frame FrG from three or moreframes, first and second interpolated frames FrH1 t and FrH2 t withpixel values at the integer coordinates of a synthesized image, and aweighting coefficient αt (x^(̂), y^(̂)) at the integer coordinates, may beacquired. In this case, for each frame FrN+t (0≦t≦T−1), pixel valuesI1N+t (x, y) in each frame FrN+t are assigned to all integer coordinatesof synthesized coordinates, and a first interpolated frame FrH1 t withpixel values I1N+t(x^(̂), y^(̂)) (i.e., I1 t(x^(̂), y^(̂)) is acquired. Byadding the pixel values I1 t(x^(̂), y^(̂)) assigned to all frames FrN+tand the pixel values I2 t (x^(̂), y^(̂)) of the second interpolated frameFrH2 t, a plurality of intermediate synthesized frames FrGt areobtained, and they are combined into a synthesized frame FrG.

More specifically, as shown in the following Formula 21, a pixel valueI1N+t(x^(̂), y^(̂)) at integer coordinates in a synthesized image iscomputed for all frames FrN+t. As shown in Formula 22, an intermediatesynthesized frame FrGt is obtained by weighting pixel values I1 t(x^(̂),y^(̂)) and I2 t(x^(̂), y^(̂)), employing a weighting coefficient α(x^(̂),y^(̂)). Further, as shown in Formula 20, a synthesized frame FrG isacquired by adding the intermediate synthesized frames FrGt.

$\begin{matrix}\begin{matrix}{{I_{{1N} + t}\left( {x^{\bigwedge},y^{\bigwedge}} \right)} = {\varphi \left( {I_{{1N} + t}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)} \right)}} \\{= {\left\{ \begin{pmatrix}{{M_{1} \times {I_{{1N} + {t\; 1}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} +} \\{{M_{2} \times {I_{{1N} + {t\; 2}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} + \ldots +} \\{{Mk} \times {I_{{1N} + {tk}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}\end{pmatrix} \right\}/}} \\{\left( {M_{1} + M_{2} + \ldots + M_{k}} \right)} \\{= \frac{\sum\limits_{i = 1}^{k}\; {M_{i} \times {I_{{1N} + {ti}}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}}}{\sum\limits_{i = 1}^{h}\; {Mi}}}\end{matrix} & (21) \\{\mspace{79mu} {{{where}\mspace{14mu} {I_{{1N} + t}\left( {{x\; {^\circ}},{y\; {^\circ}}} \right)}} = {{\prod{\left( {{Fr}_{N + t}\left( {x,y} \right)} \right).{{Fr}_{Gt}\left( {x^{\bigwedge},y^{\bigwedge}} \right)}}} = {{\alpha \; t\left( {x^{\bigwedge},y^{\bigwedge}} \right) \times {I_{1t}\left( {x^{\bigwedge},y^{\bigwedge}} \right)}} + {\left\{ {\alpha \; {t\left( {x^{\bigwedge},y^{\bigwedge}} \right)}} \right\} \times {I_{2t}\left( {x^{\bigwedge},y^{\bigwedge}} \right)}}}}}} & (22)\end{matrix}$

Note that in the case of acquiring a synthesized frame Fr_(G) from threeor more frames, three or more coordinate-transformed frames Fr_(T0) areobtained and three or more correlation values and weighting coefficientsare likewise obtained. In this case, the average or median value of theweighting coefficients may be used as a weighting coefficient for thefirst and second interpolated frames Fr_(H1) and Fr_(H2) that correspondto each other.

Now, a description will be given of a third embodiment of the presentinvention. FIG. 13 shows a video image synthesizer constructed inaccordance with the third embodiment of the present invention. Becausethe same reference numerals will be applied to the same parts as thefirst embodiment, a detailed description of the same parts will not begiven.

The third embodiment is equipped with edge information acquisition means16 instead of the correlation-value computation means 6 of the firstembodiment, and differs from the first embodiment in that, based on edgeinformation acquired by the edge information acquisition means 16,weight computation means 7 computes a weighting coefficient that is usedin weighting first and second interpolated frames Fr_(H1) and Fr_(H2).

The edge information acquisition means 16 acquires edge informatione0(x, y) that represents the edge intensity of a reference frame Fr_(N).To acquire the edge information e0(x, y), a filtering process isperformed on the reference frame Fr_(N) by employing a Laplacian filterof 3×3 shown in FIG. 14, as shown in the following Formula 23.

e0(x, y)=|∇FrN(x, y)|  (23)

In the third embodiment, a Laplacian filter is employed in the filteringprocess to acquire the edge information e0(x, y) of the reference frameFr_(N). However, any type of filter can be employed, if it is a filter,such as a Sobel filter and a Prewitt filter, which can acquire edgeinformation.

The weight computation means 7 computes a weighting coefficient α(x, y)that is used in weighting first and second interpolated frames Fr_(H1)and Fr_(H2), from the edge information e0(x, y) acquired by the edgeinformation acquisition means 6. More specifically, the weightingcoefficient α(x, y) is acquired by referring to a graph shown in FIG.15. As illustrated in the figure, the weighting coefficient α(x, y)changes linearly between the minimum value α0 and the maximum value α1.In the graph shown in FIG. 15, if the edge information e0(x, y) becomesgreater, the value of the weighting coefficient α(x, y) becomes closerto the maximum value α1. Note that the edge information e0(x, y) isrepresented by a 8-bit value.

In addition, in the synthesis means 8 of the third embodiment, if anedge intensity in the reference frame Fr_(N)becomes greater, the weightof the first interpolated frame Fr_(H1) is made greater. In this manner,the first and second interpolated frames Fr_(H1) and Fr_(H2) areweighted.

Now, a description will be given of operation of the third embodiment.FIG. 16 shows processes that are performed in the third embodiment. Inthe following description, the first interpolated frame Fr_(H1), secondinterpolated frame Fr_(H2), and weighting coefficient α(x^(∘), y^(∘))are obtained at real coordinates to which pixels in the frame Fr_(H1+1)of a synthesized image are assigned. First, as with steps S1 to S3 inthe first embodiment, steps S11 to S13 are performed.

The edge information e0(x, y) representing the edge intensity of thereference frame Fr_(N) is acquired by the edge information acquisitionmeans 16 (step S14). Based on the edge information e0(x, y), theweighting coefficient α(x^(∘), y^(∘)) is computed by the weightcomputation means 7 (step S15).

On the other hand, based on the correspondent relationship estimated bythe correspondent relationship estimation means 2, the firstinterpolated frame Fr_(H1) is acquired by spatio-temporal interpolationmeans 4 (step S16), and the second interpolated frame Fr_(H2) isacquired by spatial interpolation means 5 (step S17).

Note that the processes in steps S16 and S17 may be previously performedand the processes in steps S14 and S15 and the processes in steps S16and S17 may be performed in parallel.

In synthesis means 8, a pixel value I₁(x^(∘), y^(∘)) in the firstinterpolated frame Fr_(H1) and a pixel value I₂(x^(∘), y^(∘)) in thesecond interpolated frame Fr_(H2) are synthesized, whereby a synthesizedframe Fr_(G) consisting of a pixel value Fr_(G)(x^(̂), y^(̂)) is acquired(step S18), and the processing ends.

If the motion of a subject included in each of the frames Fr_(N) andFr_(N+1) is small, the edge intensity of the reference frame Fr_(N) willbecome great, but if the motion is great or complicated, it moves thecontour of the subject and makes the edge intensity small. Therefore, byweighting the first interpolated frame Fr_(H1) and second interpolatedframe Fr_(H2) on the basis of the weighting coefficient α(x^(∘), y^(∘))computed by the weight computation means 7, when the motion of a subjectis small there is obtained a synthesized frame Fr_(G) in which the ratioof the first interpolated frame Fr_(H1) with high definition is high,and when the motion is great there is obtained a synthesized frameFr_(G) including at a high ratio the second interpolated frame Fr_(H2)in which the blurring of a moving subject has been reduced.

Therefore, in the case where the motion of a subject included in each ofthe frames Fr_(N) and Fr_(N+1) is great, the blurring of a subject inthe synthesized frame Fr_(G) is reduced, and when the motion is small,high definition is obtained. In this manner, a synthesized frame Fr_(G)with high picture quality can be obtained independently of the motion ofa subject included in each of the frames Fr_(N) and Fr_(N+1).

In the above-described third embodiment, a synthesized frame Fr_(G) isacquired from two frames Fr_(N) and Fr_(N+1). Alternatively, it may beacquired from three or more frames, as with the above-described firstand second embodiments. In this case, the weighting coefficient α(x^(∘),y^(∘)), which is used in weighting first and second interpolated framesFr_(H1t) and Fr_(H2t) that correspond to each other, is computed basedon the edge information representing the edge intensity of the referenceframe Fr_(N).

In the above-described third embodiment, when a synthesized frame Fr_(G)is acquired from three or more frames, edge information e0(x, y) isobtained for all frames other than the reference frame Fr_(N). Becauseof this, a weighting coefficient α(x, y) is computed from the average ormedian value of many pieces of information acquired from a plurality offrames.

In the above-described third embodiment, edge information e0(x, y) isacquired from the reference frame Fr_(N) and then the weightingcoefficient α(x, y) is computed. Alternatively, the edge informatione0(x, y) may be acquired from the reference frame Fr_(N) and thesucceeding frame Fr_(N+1). In this case, assume that edge informationacquired from the reference frame Fr_(N) is e1(x, y) and edgeinformation acquired from the succeeding frame Fr_(N+1) is e2(x, y). Theaverage, multiplication, logic sum, logic product, etc., of the twopieces of information e1(x, y) and e2(x, y) are computed, and based onthem, a weighting coefficient α(x, y) is acquired.

Now, a description will be given of a fourth embodiment of the presentinvention. FIG. 17 shows a video image synthesizer constructed inaccordance with the fourth embodiment of the present invention. Becausethe same reference numerals will be applied to the same parts as thefirst embodiment, a detailed description of the same parts will not begiven. The fourth embodiment is provided with sampling means 11 andcorrespondent relationship acquisition means 12 instead of the samplingmeans 1 and correspondent relationship estimation means 2 of the firstembodiment, and is further equipped with stoppage means 10 for stoppinga process that is performed in the correspondent relationshipacquisition means 12. The fourth embodiment differs from the firstembodiment in that, for a plurality of frames to be stopped by thestoppage means 10, a correspondent relationship between a pixel in areference frame and a pixel in each of the frames other than thereference frame is acquired in order of other frames closer to thereference frame by the correspondent relationship acquisition means 12.Note that in the fourth embodiment, coordinate transformation means 3,spatio-temporal interpolation means 4, spatial interpolation means 5,correlation-value computation means 6, weight computation means 7, andsynthesis means 8 as a whole constitute frame synthesis meanshereinafter claimed.

FIG. 18 shows the construction of the sampling means 11 of the videoimage synthesizer shown in FIG. 17. As illustrated in FIG. 18, thesampling means 11 is equipped with storage means 22, condition settingmeans 24, and sampling execution means 26. The storage means 22 is usedto store a frame-number determination table, in which magnificationratios of a pixel size in a synthesized frame to a pixel size in oneframe of a video image, video image frame rates, and compressionqualities, and frame numbers S are caused to correspond to one another.The condition setting means 24 is used for inputting a magnificationratio of a pixel size in a synthesized frame Fr_(G) to a pixel size inone frame of a video image, and a frame rate and compression quality forvideo image data M0. The sampling execution means 26 refers to theframe-number determination table stored in the storage means 22, thendetects the frame number S corresponding to the magnification ratio,frame rate, and compression quality input through the condition settingmeans 24, and samples S contiguous frames from video image data M0.

FIG. 19 shows an example of the frame-number determination table storedin the storage means 22 of the sampling means 11 shown in FIG. 18. Inthe illustrated example, frame number S to be sampled is computed fromvarious combinations of a magnification ratio, frame rate, andcompression quality in accordance with the following Formula 24.

S=min (S1, S2×S3)   (24)

S1=frame rate×3

S2=magnification ratio×1.5

S3=1.0 (high compression quality)

S3=1.2 (intermediate compression quality)

S3=1.5 (low compression quality)

That is, if the frame rate is great the frame number S is increased, ifthe magnification ratio is great the frame number S is increased, and ifthe compression quality is low the frame number S is increased. In thistendency, the number of frames is determined.

The sampling means 11 outputs S frames sampled to the correspondentrelationship acquisition means 12, in which correspondent relationshipsbetween a pixel in a reference frame of the S frames (when theprocessing of a frame is stopped by the stoppage means 10, frames up tothe stopped frame) and a pixel in each of the frames other than thereference frame are acquired in order of other frames closer to thereference frame. The video image data M0 represents a color video image,and each frame consists of a luminance component Y and two colordifference components Cb and Cr. In the following description, processesare performed on the three components, but are the same for eachcomponent. Therefore, in the fourth embodiment, a detailed descriptionwill be given of processes that are performed on the luminance componentY, and a description of processes that are performed on the colordifference components Cb and Cr will not be made.

In the S frames output from the sampling means 11, for example, thefirst frame is the reference frame Fr_(N), and frames Fr_(N+1),Fr_(N+2), . . . , and Fr_(N+(S−1)) are contiguously arranged in ordercloser to the reference frame.

The correspondent relationship acquisition means 12 acquires acorrespondent relationship between the frames Fr_(N) and Fr_(N+1) by thesame process as the process performed in the correspondent relationshipestimation means 2 of the above-described first embodiment.

For the S frames output from the sampling means 11, the correspondentrelationship acquisition means 12 acquires correspondent relationshipsin order closer to the reference frame Fr_(N), but when the processingof a frame is stopped by the stoppage means 10, the acquisition of acorrespondent relationship after the stopped frame is stopped.

FIG. 20 shows the construction of the stoppage means 10. As shown in thefigure, the stoppage means 10 is equipped with correlation acquisitionmeans 32 and stoppage execution means 34. The correlation acquisitionmeans 32 acquires a correlation between a frame being processed by thecorrespondent relationship acquisition means 12 and the reference frame.If the correlation acquired by the correlation acquisition means 32 is apredetermined threshold value or greater, the processing in thecorrespondent relationship acquisition means 12 is not stopped. If thecorrelation is less than the predetermined threshold, the acquisition ofa correspondent relationship after a frame being processed by thecorrespondent relationship acquisition means 12 is stopped by thestoppage execution means 34.

In the fourth embodiment, the sum of correlation values E at the time ofconvergence, computed from one frame by the correspondent relationshipacquisition means 12, is employed as a correlation value between the oneframe and the reference frame by the correlation acquisition means 32,and if this correlation value is a predetermined threshold value orgreater (that is, if correlation is a predetermined threshold value orless), the processing in the correspondent relationship acquisitionmeans 12 is stopped, that is, the acquisition of a correspondentrelationship after a frame being processed is stopped.

The frame synthesis means, which consists of coordinate transformationmeans 3, etc., acquires a synthesized frame in the same manner as theabove-described first embodiment, employing the reference frame andother frames (in which correspondent relationships with the referenceframe have been acquired), based on the correspondent relationshipacquired by the correspondent relationship acquisition means 12).

FIG. 21 shows processes that are performed in the fourth embodiment. Inthis embodiment, consider the case where a first interpolated frameFr_(H1), a second interpolated frame Fr_(H2), and a weightingcoefficient α(x^(∘), y^(∘)) are acquired at real coordinates to whichpixels of the frame Fr_(N+1) in a synthesized image are assigned. In thevideo image synthesizer of the fourth embodiment, as shown in FIG. 21,video image data M0 is first input (step S22). To acquire a synthesizedframe from the video image data M0, a magnification ratio, frame rate,and compression quality are input through the condition setting means 24of the sampling means 11 (step S24). The sampling execution means 26refers to the frame-number determination table stored in the storagemeans 22, then detects the frame number S corresponding to themagnification ratio, frame rate, and compression quality input throughthe condition setting means 24, and samples S contiguous frames fromvideo image data M0 and outputs them to the correspondent relationshipacquisition means 12 (step S26). The correspondent relationshipacquisition means 12 places a reference patch on the reference frameFr_(N) of the S frames (step S28), also places the same patch as thereference patch on the succeeding frame Fr_(N+1), and moves and/ordeforms the patch until a correlation value E with an image within thereference patch converges (steps S32 and S34). In the stoppage means 10,the sum of correlation values E at the time of convergence is computed.If the sum is a predetermined threshold value or greater (that is, ifthe correlation between this frame and the reference frame is thepredetermined threshold value or less), the processing in thecorrespondent relationship acquisition means 12 is stopped. That is, bystopping the acquisition of a correspondent relationship after thestopped frame, the processing in the video image synthesizer is shiftedto processes that are performed in the frame synthesis means (consistingof coordinate transformation means 3, etc.) (“NO” in step S36, steps S50to S60).

On the other hand, if the processing in the correspondent relationshipacquisition means 12 is not stopped by the stoppage means 10, thecorrespondent relationship acquisition means 12 acquires correspondentrelationships between the reference frame and the (S−1) frames excludingthe reference frame and outputs the correspondent relationships to theframe synthesis means (“NO” in step S36, step S38, “YES” in step S40,step S45).

Steps S50 to S60 show operation of the frame synthesis means consistingof coordinate transformation means, etc. For convenience, a descriptionwill be given in the case where the correspondent relationshipacquisition means 12 acquires only a correspondent relationship betweenthe reference frame Fr_(N) and the succeeding frame Fr_(N+1).

Based on the correspondent relationship acquired by the correspondentrelationship acquisition means 12, the coordinate transformation means 3transforms the coordinates of the succeeding frame Fr_(N+1) to acoordinate space in the reference frame Fr_(N) and acquires acoordinate-transformed frame Fr_(T0) (step S50). Next, thecorrelation-value computation means 6 computes the correlation valued0(x, y) between the coordinate-transformed frame Fr_(T0) and thereference frame Fr_(N) (step S52). Based on the correlation value d0(x,y), the weight computation means 7 computes a weighting coefficientα(x^(∘), y^(∘)) (step S54).

On the other hand, based on the correspondent relationship acquired bythe correspondent relationship acquisition means 12, the spatio-temporalinterpolation means 4 acquires a first interpolated frame Fr_(H1) (stepS56), and the spatial interpolation means 5 acquires a secondinterpolated frame Fr_(H2) (step S58).

Note that the processes in steps S56 to S58 may be previously performedand the processes in steps S50 to S54 and the processes in steps S56 toS58 may be performed in parallel.

In the synthesis means 8, a pixel value I₁(x^(∘), y^(∘)) in the firstinterpolated frame Fr_(H1) and a pixel value I₂(x^(∘), y^(∘)) in thesecond interpolated frame Fr_(H2) are synthesized, whereby a synthesizedframe Fr_(G) consisting of a pixel value Fr_(G)(x^(̂), y^(̂)) is acquired(step S60), and the processing ends.

In the fourth embodiment, for the convenience of explanation, thecorrespondent relationship acquisition means 12 acquires only acorrespondent relationship between the reference frame Fr_(N) and thesucceeding frame Fr_(N+1), and the frame synthesis means obtains asynthesized frame from the two contiguous frames. For instance, in thecase of acquiring a synthesized frame Fr_(G) from T (T≧3) framesFr_(N+t′) (0≦t′≦T−1) (that is, in the case where the correspondentrelationship acquisition means 12 acquires two correspondentrelationships between the reference frame Fr_(N) and two contiguousframes), pixel values are assigned to a synthesized image, and aplurality of first interpolated frames Fr_(H1t) are obtained for thecontiguous frames Fr_(N+t) (0≦t≦T−1) other than the reference frameFr_(N)(=Fr_(N+0)). Note that a pixel value in the first interpolatedframe Fr_(H1t) is represented by I_(1t)(x^(∘), y^(∘)).

Thus, in the video image synthesizer of the fourth embodiment, thesampling means 11 determines the number of frames to be sampled, basedon the compression quality and frame rate of the video image data M0 andon the magnification ratio of a pixel size in a synthesized frame to apixel size in a frame of a video image. Therefore, the operator does notneed to determine the number of frames, and the video image synthesizercan be conveniently used. By determining the number of frames on thebasis of image characteristics between a video image and a synthesizedframe, a suitable number of frames can be objectively determined, so asynthesized frame with high quality can be acquired.

In addition, in the video image synthesizer of the fourth embodiment,for S frames sampled, a correspondent relationship between a pixelwithin a reference patch on the reference frame and a pixel within apatch on the succeeding frame is computed in order of other framescloser to the reference frame, and a correlation between the referenceframe and the succeeding frame is obtained. If the correlation is apredetermined threshold value or greater, then a correspondentrelationship with the next frame is acquired. On the other hand, if aframe whose correlation is less than the predetermined threshold valueis detected, the acquisition of correspondent relationships with otherframes after the detected frame is stopped, even when the number offrames does not reach the determined frame number. This can avoidacquiring a synthesized frame from a reference frame and a frame whosecorrelation is low (e.g., a reference frame for a scene and a frame fora switched scene), and makes it possible to acquire a synthesized frameof higher quality.

Note that in the fourth embodiment, the stoppage means 10 stops theprocesses of the correspondent relationship acquisition means 12 in thecase that the sum of E is higher than a predetermined threshold value.However, the stoppage means may also stop the processes of the framesynthesis means as well.

Now, a description will be given of a fifth embodiment of the presentinvention. FIG. 22 shows a video image synthesizer constructed inaccordance with the fifth embodiment of the present invention. Since thesame reference numerals will be applied to the same parts as the fourthembodiment, a detailed description of the same parts will not be given.The fifth embodiment is equipped with sampling means 11A instead of thesampling means 11 of the fourth embodiment, and differs from the fourthembodiment in that it does not include the above-described stoppagemeans 10.

FIG. 23 shows the construction of the sampling means 11A of the videoimage synthesizer shown in FIG. 22. As illustrated in FIG. 23, thesampling means 11A is equipped with reduction means 42, correlationacquisition means 44, stoppage means 46, and sampling execution means48. The reduction means 42 performs a reduction process on video imagedata M0 to obtain reduced video image data. For the reduced video imagedata obtained by the reduction means 42, the correlation acquisitionmeans 44 acquires a correlation between a reduction reference frame(which is discriminated from the reference frame in the video image dataM0) and each of the succeeding reduction frames (which are discriminatedfrom the contiguous frames in the video image data M0). The stoppagemeans 46 monitors the number of reduction frames whose correlation hasbeen obtained by the correlation acquisition means 44, and stops theprocessing in the correlation acquisition means 44 when the frame numberreaches a predetermined upper limit value. When the processing in thecorrelation acquisition means 44 is not stopped by the stoppage means46, the sampling execution means 48 sets a sampling range on the basisof a correlation between adjacent reduction frames acquired by thecorrelation acquisition means 44, and samples frames from the videoimage data M0 in a range corresponding to the sampling range. Thesampling range is from the reduction reference frame to a reductionframe, which is closer to the reduction reference frame, between a pairof adjacent reduction frames whose correlation is lower than apredetermined threshold value. On the other hand, when the processing inthe correlation acquisition means 44 is stopped by the stoppage means46, the sampling execution means 48 sets a sampling range from areduction reference frame to a reduction frame being processed at thetime of the stoppage, and samples frames from the video image data M0 ina range corresponding to the sampling range. Note that when acquiring acorrelation between adjacent reduction frames, with a reductionreference frame as the first frame, a correlation between reductionframes adjacent after the reduction reference frame may be acquired.Also, with a reduction reference frame as the last frame, a correlationbetween reduction frames adjacent before the reduction reference framemay be acquired. Furthermore, a correlation between reduction framesadjacent before a reduction reference frame, and a correlation betweenreduction frames adjacent after the reduction reference frame, may beacquired and the aforementioned sampling range may include the reductionreference frame. In the fifth embodiment, a sampling range is detectedwith a reduction reference frame as the first frame.

The correlation acquisition means 44 in the fifth embodiment computes ahistogram for the luminance component Y of each reduction frame, alsocomputes a Euclidean distance between adjacent reduction framesemploying the histogram, and employs the distance as a correlation valuebetween adjacent reduction frames. When the processing in thecorrelation acquisition means 44 is not stopped by the stoppage means46, the sampling execution means 48 sets a sampling range on the basisof a correlation between adjacent reduction frames acquired by thecorrelation acquisition means 44, and samples frames from the videoimage data M0 in a range corresponding to the sampling range. Thesampling range is from the reduction reference frame to a reductionframe, which is closer to the reduction reference frame, between a pairof adjacent reduction frames whose correlation is lower than apredetermined threshold value (that is, a correlation value consistingof the Euclidean distance is higher than a predetermined thresholdvalue). On the other hand, when the processing in the correlationacquisition means 44 is stopped by the stoppage means 46, the samplingexecution means 48 sets a sampling range from a reduction referenceframe to a reduction frame being processed at the time of the stoppage,and samples frames from the video image data M0 in a range correspondingto the sampling range.

The sampling means 11A outputs a plurality of frames (S frames) to thecorrespondent relationship acquisition means 12, which acquires acorrespondent relationship between a pixel in a reference frame of the Sframes and a pixel in the succeeding frame.

FIG. 24 shows processes that are performed in the fifth embodiment. Aswith the fourth embodiment, consider the case where a first interpolatedframe FrH1, a second interpolated frame FrH2, and a weightingcoefficient α(x^(∘), y^(∘)) are acquired at real coordinates to whichpixels of the frame FrN+1 in a synthesized image are assigned. In thevideo image synthesizer of the fifth embodiment, as shown in FIG. 24,video image data M0 is first input (step S62). To acquire a synthesizedframe from the video image data M0, the reduction means 42 of thesampling means 11A performs a reduction process on the video image dataM0 and obtains reduced video image data (step S64). The samplingexecution means 48 sets a sampling range on the basis of a correlationbetween each reduction frame and a reduction reference frame acquired bythe correlation acquisition means 44, and samples frames from the videoimage data M0 in a range corresponding to the sampling range. Thesampling range is from the reduction reference frame to a reductionframe, which is closer to the reduction reference frame, between a pairof adjacent reduction frames whose correlation is lower than apredetermined threshold value. On the other hand, when the processing inthe correlation acquisition means 44 is stopped by the stoppage means46, the sampling execution means 48 sets a sampling range from areduction reference frame to a reduction frame being processed at thetime of the stoppage, and samples frames from the video image data M0 ina range corresponding to the sampling range. The S frames sampled by thesampling execution means 48 are output to the correspondent relationshipacquisition means 12 (step S66). The correspondent relationshipacquisition means 12 places a reference patch on the reference frame FrN(step S68), also places the same patch as the reference patch on thesucceeding frame FrN+1, and moves and/or deforms the patch until acorrelation value E between an image within the reference patch and animage within the patch of the succeeding frame FrN+1 converges (stepsS72 and S74). The correspondent relationship acquisition means 12acquires a correspondent relationship between the reference frame FrNand the succeeding frame FrN+1 (step S78). The correspondentrelationship acquisition means 12 performs the processes in steps S72 toS78 on all frames excluding the reference frame (“YES” in step S80, stepS85).

The processes in steps S90 to S100 correspond to the processes in stepsS50 to S60 of the fourth embodiment.

In the above-described fifth embodiment, a synthesized frame Fr_(G) isacquired from two frames Fr_(N) and Fr_(N+1). Alternatively, it may beacquired from three or more frames, as with the above-described fourthembodiment.

Thus, in the video image synthesizer of the fifth embodiment, thesampling means 11A detects a plurality of frames representing successivescenes as a contiguous frame group when acquiring a synthesized framefrom a video image, and acquires the synthesized frame from this framegroup. Therefore, the operator does not need to sample frames manually,and the video image synthesizer can be conveniently used. In addition, aplurality of frames within the contiguous frame group represent scenesthat have approximately the same contents, so the video imagesynthesizer is suitable for acquiring a synthesized frame of highquality.

In addition, in the video image synthesizer of the fifth embodiment,there is provided a predetermined upper limit value. In detecting acontiguous frame group, the detection of frames is stopped when thenumber of frames in that contiguous frame group reaches thepredetermined upper limit value. This can avoid employing a great numberof frames wastefully when acquiring one synthesized frame, and makes itpossible to perform processing efficiently.

In the fifth embodiment, although the correlation acquisition means 44of the sampling means 11A computes a Euclidean distance for a luminancecomponent Y between two adjacent reduction frames as a correlationvalue, it may also compute three Euclidean distances for a luminancecomponent Y and two color difference components Cb and Cr to employ thesum of the three Euclidean distances as a correlation value.Alternatively, by computing a difference in pixel value betweencorresponding pixels of adjacent reduction frames, the sum of absolutevalues of the pixel value differences may be employed as a correlationvalue.

Further, in computing a Euclidean distance for a luminance component Y(or the sum of three Euclidean distance for a luminance component Y andtwo color difference components Cb and Cr) as a correlation value,expedient processing may be achieved by dividing the luminance componentY (or three components Y, Cb, and Cr) by a value greater than 1 andacquiring a histogram.

In the fifth embodiment, although the correlation acquisition means 44of the sampling means 11A computes a correlation value employing thereduced video image data of the video image data M0, it may also employthe video image data M0 itself, or video image data obtained by thinningthe video image data M0.

Now, a description will be given of a sixth embodiment of the presentinvention. FIG. 25 shows a video image synthesizer constructed inaccordance with the sixth embodiment of the present invention. Since thesame reference numerals will be applied to the same parts as the fourthembodiment, a detailed description of the same parts will not be given.The sixth embodiment is equipped with sampling means 11B instead of thesampling means 11 of the fourth embodiment. The sampling means 11Bextracts a frame group constituting one or more important scenes frominput video image data M0, and also determines one reference frame froma plurality of frames constituting that frame group. The sixthembodiment differs from the fourth embodiment in that it does notinclude the aforementioned stoppage means 10 and that correspondentrelationship acquisition means 12 acquires a correspondent relationshipbetween a pixel in the reference frame of each frame group extracted bythe sampling means 11B and a pixel in a frame other than the referenceframe.

FIG. 26 shows the construction of the sampling means 11B of the videoimage synthesizer shown in FIG. 25. As illustrated in FIG. 26, thesampling means 11B is equipped with image-type input means 52,extraction control means 54, first extraction means 56, secondextraction means 58, and reference-frame determination means 60. Theimage-type input means 52 inputs a designation of either an “ordinaryimage” or a “security camera image” to indicate the type of video imagedata M0. The extraction control means 54 controls operation of the firstextraction means 56 and second extraction means 58. The first extractionmeans 56 computes a correlation between adjacent frames in the videoimage data M0, extracts as a first frame group a set of contiguousframes whose correlation is high, and outputs the first frame group tothe reference-frame determination means 60 or to second extraction means58. The second extraction means 58 computes a correlation between centerframes of the first frame groups extracted by the first extraction means56 and extracts the first frame group interposed between two first framegroups whose correlation is high and which are closest to each other, asa second frame group. The reference-frame determination means 60determines the center frame of each frame group output by the firstextraction means 56 or second extraction means 58, as a reference framefor that frame group.

When the type of video image data M0 input by the image-type input means52 is an ordinary image, the extraction control means 54 causes thefirst extraction means 56 to extract first frame groups and output theextracted first frame groups to the reference-frame determination means60. On the other hand, when the type of video image data M0 input by theimage-type input means 52 is a security camera image, the extractioncontrol means 54 causes the first extraction means 56 to extract firstframe groups and output the extracted first frame groups to the secondextraction means 58, and also causes the second extraction means 58 toextract second frame groups from the first frame groups and output themto the reference-frame determination means 60.

FIG. 27A shows the construction of the first extraction means 56 in thesampling means 11B shown in FIG. 26; FIG. 27B shows a frame groupextracted from the video image data M0 by the first extraction means 56.

As shown in FIG. 27A, the first extraction means 56 is equipped withfirst correlation computation means 72 for computing a correlationbetween adjacent frames of the video image data M0, and first samplingexecution means 74 for extracting as a first frame group a set of frameswhose correlation is high. The first correlation computation means 72computes a histogram for the luminance component Y of each frame of thevideo image data M0, also computes a Euclidean distance between adjacentframes employing this histogram, and employs the Euclidean distance as acorrelation value between frames. Based on the correlation value betweenadjacent frames acquired by the first correlation computation means 72,the first sampling execution means 74 extracts a set of contiguousframes whose correlation value is smaller than a predetermined thresholdvalue (that is, the correlation is higher than the predeterminedthreshold value), as a first frame group. For example, a plurality offirst frame groups G1 to G7 are extracted as shown in FIG. 27B.

FIG. 28 shows the construction of the second extraction means 58 in thesampling means 11B shown in FIG. 26. The second extraction means 58extracts second frame groups from the first frame groups extracted bythe first extraction means 56, when the video image data is a securitycamera image. As illustrated in FIG. 28, the second extraction means 58is equipped with second correlation computation means 76 and secondsampling execution means 78. With respect to the first frame groupsextracted by the first extraction means 56 (e.g., G1, G2 . . . G7 inFIG. 27B), the second correlation computation means 76 computes aEuclidean distance for the luminance component Y between center framesof the first frame groups not adjacent (e.g. center frames of G1 and G3,G1 and G4, G1 and G5, G1 and G6, G1 and G7, G2 and G4, G2 and G5, G2 andG6, G2 and G7, . . . , G4 and G6, G4 and G7, and G5 and G7 in FIG. 27B),and employs the Euclidean distance between center frames as acorrelation value between the first frame groups to which the centerframes belong. Based on each correlation value acquired by the secondcorrelation computation means 76, the second sampling execution means 78extracts the first frame group interposed between two first frame groupswhose correlation value is smaller than a predetermined threshold value(that is, correlation is higher than the predetermined threshold value)and which are closest to each other, as a second frame group. Forexample, in the first frame groups shown in FIG. 27A, if (G1 and G3) and(G4 and G7) are first frame groups whose correlation is high and whichare closest to each other, G2 between G1 and G3 and (G5+G6) between G4and G7 are extracted as second frame groups.

Now, a description will be given of characteristics of the first andsecond frame groups. When picking up an image, there is a tendency topick up an interesting scene for a relatively long time (e.g., a fewseconds) without moving a camera, so frames having approximately thesame contents for a relatively long time can be considered to be animportant scene in ordinary video image data. That is, the firstextraction means 56 of the sampling means 11B of the video imagesynthesizer shown in FIG. 25 is used to extract important scenes fromthe video image data of an ordinary image.

On the other hand, in the case of a video image (security camera image)taken by a security camera, different scenes for a short time (e.g.,scenes picking up an intruder), included in scenes of the same contentswhich continues for a long time, can be considered important scenes.Therefore, a second frame group, extracted by the second extractionmeans 58 of the sampling means 11B of the video image synthesizer shownin FIG. 25, can be considered a frame group that represents an importantscene in the case of a security camera image.

With respect to the first frame groups output from the first extractionmeans 56 or second frame groups output from the second extraction means58, the reference-frame determination means 60 of the sampling means 11Bdetermines the center frame of each frame group as the reference frameof the frame group, and also outputs each frame group to the framesynthesis means along with information representing a reference frame.In the case where a second frame group consists of a plurality of firstframe groups, like the aforementioned example (G5 and G6), the centerframe of all frames included in the second frame group is employed asthe center frame of the second frame group.

With respect to the frame groups output from the sampling means 11B, thecorrespondent relationship acquisition means 12 and frame synthesismeans acquire a synthesized frame Fr_(G) for each frame group, and theprocess of acquiring a synthesized frame Fr_(G) is the same in eachframe group, so a description will be given of the process of acquiringa synthesized frame from one frame group by the correspondentrelationship acquisition means 12 and frame synthesis means.

With respect to one frame group (which consists of T frames) output fromthe sampling means 11B, the correspondent relationship acquisition means12 acquires a correspondent relationship between a pixel in a referenceframe of the T frames and a pixel in each of the (T−1) frames other thanthe reference frame. Note that the correspondent relationshipacquisition means 12 acquires a correspondent relationship between thereference frame Fr_(N) and the succeeding frame Fr_(N+1) by the sameprocess as the process performed in the correspondent relationshipacquisition means 2 of the above-described first embodiment.

FIG. 29 shows processes that are performed in the sixth embodiment. Inthe video image synthesizer of the sixth embodiment, as shown in FIG.29, video image data M0 is first input (step S102). Based on the imagetype (ordinary image or security camera image) of the video image dataM0 input through the image-type input means 52, the extraction controlmeans 54 controls operation of the first extraction means 56 or secondextraction means 58 to extract a frame group that constitutes animportant scene (steps S104 to S116). More specifically, if the imagetype of video image data M0 is an ordinary image (“YES” in step S106),the extraction control means 54 causes the first extraction means 56 toextract first frame groups and output them to the reference-framedetermination means 60 as frame groups that constitute an importantscene (step S108). On the other hand, if the video image data M0 is asecurity camera image (“NO” in step S106), the extraction control means54 causes the first extraction means 56 to extract first frame groupsand output them to the second extraction means 58 (step S110), and alsocauses the second extraction means 58 to extract second frame groupsfrom the first frame groups extracted by the first extraction means 56and output the extracted second frame groups to the reference-framedetermination means 60 as frame groups that constitute an importantscene in the video image data M0 (step S112).

With respect to the first frame groups output from the first extractionmeans 56 or second frame groups output from the second extraction means58, the reference-frame determination means 60 determines the centerframe of each frame group as the reference frame of the frame group, andalso outputs each frame group to the correspondent relationshipacquisition means 12 and frame synthesis means along with informationrepresenting a reference frame (step S114).

The correspondent relationship acquisition means 12 acquires acorrespondent relationship between a reference frame and a frame otherthan the reference frame, for each frame group. Based on thecorrespondent relationship obtained by the correspondent relationshipacquisition means 12, the frame synthesis means (which consists ofspatio-temporal interpolation means 4, etc.) acquires a synthesizedframe for each frame group with respect to all frame groups output fromthe sampling means 11B (steps S116, S118, S120, S122, and S124).

Thus, in the video image synthesizer of the sixth embodiment, thesampling means 11B extracts frame groups constituting an important scenefrom video image data M0 and determines the center frame of a pluralityof frames constituting each frame group, as the reference frame of theframe group. Therefore, the operator does not need to set a referenceframe manually, and the video image synthesizer can be convenientlyused. In sampling a plurality of frames, unlike a method of setting areference frame and then sampling frames in a range including thereference frame, frames constituting an important scene included invideo image data are extracted and then a reference frame is determinedso that a synthesized frame is obtained for each important scene. Thus,the intention of an photographer can be reflected.

Further, the video image synthesizer of the sixth embodiment areequipped with two extraction means so that, based on the type of videoimage data (e.g., the purpose for which video image data M0 is used), animportant scene coinciding with the type can be extracted. Thus,synthesized frames, which coincide with the purpose of an photographer,can be obtained efficiently. For instance, in the case of ordinaryimages, synthesized frames can be obtained for each scene interesting toan photographer. In the case of security camera images, synthesizedframes can be obtained for only scenes required for preventing crimes.

FIG. 30 shows a video image synthesizer constructed in accordance with aseventh embodiment of the present invention. The same reference numeralswill be applied to the same parts as the sixth embodiment, so a detaileddescription of the same parts will not be given.

As illustrated in the figure, the video image synthesizer of the seventhembodiment differs from the sixth embodiment in that it is equipped withsampling means 11C instead of the sampling means 11B in the video imagesynthesizer of the sixth embodiment. The sampling means 11C of theseventh embodiment extracts a frame group constituting one or moreimportant scenes from input video image data M0, and also determines onereference frame from a plurality of frames constituting each framegroup.

FIG. 31 shows the construction of the sampling means 11C of the videoimage synthesizer shown in FIG. 30. As illustrated in FIG. 31, thesampling means 11C of the video image synthesizer of the seventhembodiment has the same construction as that of the sampling means 11Bof the video image synthesizer of the sixth embodiment exceptreference-frame determination means (60, 60′).

With respect to each frame group output from first extraction means 56or second extraction means 58, the reference-frame determination means60′ of the sampling means 11C of the video image synthesizer of theseventh embodiment determines a frame that is most in focus among aplurality of frames constituting a frame group, as the reference frameof that frame group. More specifically, to determine the reference frameof one frame group, the high-frequency components of frames constitutingthat frame group are extracted, the sum total of high-frequencycomponents is computed for each frame, and a frame whose sum total ishighest is determined as the reference frame of that frame group. Notethat a method of extracting high-frequency components may be any methodthat is capable of extracting high-frequency components. For instance, adifferential filter or Laplacian filter may be employed, or Wavelettransformation may be performed.

According to the video image synthesizer of the seventh embodiment, thesame advantages as the video image synthesizer of the sixth embodimentcan be obtained, and when picking up images, a frame that is most infocus is determined as a reference frame by taking advantage of the factthat a camera is often focused on an important scene. This is able tomake a contributory degree to the acquisition of synthesized frames ofhigh quality.

In computing a correlation value, the first correlation acquisitionmeans 72 and second correlation acquisition means 76 of the samplingmeans 11B and sampling means 11C in the video image synthesizers of theabove-described sixth and seventh embodiments compute a Euclideandistance for a luminance component Y between two frames as a correlationvalue. However, by computing three Euclidean distances for a luminancecomponent Y and two color difference components Cb and Cr, the sum ofthe three Euclidean distances maybe employed as a correlation value.Also, by computing a difference in pixel value between correspondingpixels of two frames, the sum of absolute values of the pixel valuedifferences may be employed as a correlation value.

Further, expedient processing may be achieved by employing the videoimage data M0 itself, or video image data obtained by thinning the videoimage data M0, when computing a correlation.

In the above-described sixth and seventh embodiments, a synthesizedframe Fr_(G) is acquired from two frames Fr_(N) and Fr_(N+1).Alternatively, it may be acquired from three or more frames, as in theabove-described fourth embodiment.

Now, a description will be given of an eighth embodiment of the presentinvention. FIG. 32 shows an image processor constructed in accordancewith the eighth embodiment of the present invention. As illustrated inthe figure, the image processor of the eighth embodiment of the presentinvention is equipped with sampling means 101, similarity computationmeans 102, contributory degree computation means 103, and synthesismeans 104. The sampling means 101 samples a plurality of frames Fr₁, Fr₂. . . Fr_(N) from video image data M0. The similarity computation means102 computes similarities b2, b3 . . . bn between one frame to beprocessed (e.g., frame Fr₁) and other frames Fr₂ . . . Fr_(N). Based onthe similarities computed by the similarity computation means 102, thecontributory degree computation means 103 computes contributory degrees(i.e., weighting coefficients) β1, β2 . . . βn that are employed inweighting the frames Fr₂ . . . Fr_(N) and adding the weighted frames tothe frame Fr₁. In accordance with the contributory degrees β1, β2 . . .βn, the synthesis means 104 weights the frames Fr₂ . . . Fr_(N) and addsthe weighted frames to the frame Fr₁ and acquires a processed frameFr_(G).

The sampling means 101 samples frames Fr₁, Fr₂ . . . Fr_(N) from videoimage data M0 at equal temporal intervals. In the eighth embodiment,three frames Fr₁, Fr₂, and Fr₃ temporally adjacent are employed andframes Fr₂ and Fr₃ are weighted and added to frame Fr₁.

The similarity computation means 102, as shown in FIG. 33, performs theparallel movement or affine transformation of Fr₁ with respect to frameFr₂ and frame Fr₃. When the correlation between a pixel value in frameFr₁ and a pixel value in frame Fr₂ or Fr₃ is highest, the accumulationof the square of a difference between pixel values in frame Fr₁ andframe Fr₂ and square of a difference between pixel values in frame Fr₁and frame Fr₃, or the reciprocal of the accumulation of absolute values,are computed as similarities b2 and b3.

Note that a correlation between corresponding pixels becomes highestwhen the accumulation of the square of differences between pixel valuesin frame Fr₁ and frames Fr₂ and Fr₃ or the reciprocal of theaccumulation of absolute values becomes smallest. Therefore,similarities b2 and b3 have a great value if frames Fr₂ and Fr₃ aresimilar to frame Fr₁. In FIG. 33, when a subject Q0 in frame Fr₁coincides with a subject Q0 in frame Fr₂ or Fr₃, the correlation betweena pixel value in frame Fr₁ and a pixel value in frame Fr₂ or Fr₃ becomeshighest.

The contributory degree computation means 103 computes contributorydegrees β2 and β3, which are employed in weighing frames Fr₂ and Fr₃ andadding to frame Fr₁, by multiplying similarities b2 and b3 by apredetermined reference contributory degree k.

The synthesis means 104 acquires a processed frame Fr_(G) by weightingframes Fr₂ and Fr₃ and adding to frame Fr₁, in accordance withcontributory degrees β2 and β3. More specifically, if frame datarepresenting frames Fr₁, Fr₂, and Fr₃ are S1, S2, and S3, and frame datarepresenting a processed frame Fr_(G) is SG, the processed frame data SGis computed by the following Eq. 25.

SG=S1+β2·S2+β3·S3   (25)

For example, in the case where frame Fr₂ has a pixel size of 4×4, eachpixel has a value shown in FIG. 34A, and contributory degree β2 is 0.1,a pixel value of each pixel in frame Fr₂ that is added to frame Fr₁ isone-tenth a value shown in FIG. 34A, as shown in FIG. 34B.

Note that frame data S1, S2, and S3 may be red, green, and blue data,respectively. They may also be luminance data and color difference data,or may be only luminance data.

Now, a description will be given of operation of the eighth embodiment.FIG. 35 shows processes that are performed in the eighth embodiment.First, the sampling means 101 samples frames Fr₁, Fr₂, and Fr₃ fromvideo image data M0 (step S131). Then, in the similarity computationmeans 102, similarities b2 and b3 between frame Fr₁ and frames Fr₂, Fr₃are computed (step S132). In the contributory degree computation means103, contributory degrees β2 and β3 are computed by multiplyingsimilarities b2 and b3 by a reference contributory degree k (step S133).Next, in accordance with contributory degrees β2 and β3, frames Fr₂ andFr₃ are weighted and added to frame Fr₁, whereby a processed frameFr_(G) is obtained (step S134) and the processing ends.

Thus, in the eighth embodiment, with respect to frames Fr2 and Fr3temporally before and after frame Fr1, similarities b2 and b3 with frameFr1 are computed, and if similarities b2 and b3 are great, contributorydegrees (weighting coefficients) β2 and β3 are made greater. Frames Fr2and Fr3 are weighted and added to frame Fr1, whereby a processed frameFrG is obtained. Because of this, there is no possibility that a framenot similar to frame Fr1, as it is, will be added to frame Fr1. Thisrenders it possible to add frames Fr2 and Fr3 to frame Fr1 whilereducing the influence of dissimilar frames. Consequently, a processedframe FrG with high quality can be obtained while reducing blurring thatis caused by synthesis of frames whose similarity is low.

In the above-described eighth embodiment, although a processed frameFr_(G) is obtained by multiplying frames Fr₂ and Fr₃ by contributorydegrees β2 and β3 and adding the weighted frames to frame Fr₁, aprocessed frame Fr_(G) with higher resolution than frame Fr₁ may beobtained by interpolating frames Fr₂ and Fr₃ multiplied by contributorydegrees β2 and β3 in frame Fr₁, like a method disclosed in JapaneseUnexamined Patent Publication No. 2000-354244, for example.

Now, a description will be given of a ninth embodiment of the presentinvention. FIG. 36 shows an image processor constructed in accordancewith the ninth embodiment of the present invention. In the ninthembodiment, the same reference numerals will be applied to the sameparts as the eighth embodiment, so a detailed description of the sameparts will not be given. As shown in FIG. 36, the image processor of theninth embodiment is equipped with similarity computation means 112,contributory degree computation means 113, and synthesis means 114,instead of the similarity computation means 102, contributory degreecomputation means 103, and synthesis means 104 of the eighth embodiment.The similarity computation means 112 partitions frame Fr₁ into m×nblock-shaped areas A1(m, n) and computes similarities b2(m, n) and b3(m,n) for areas A2(m, n) and A3(m, n) in frames Fr₂ and Fr₃ whichcorrespond to area A1(m, n). The contributory degree computation means113 computes contributory degrees β2(m, n) and β3(m, n) for areas A2(m,n) and A3(m, n). In accordance with the computed contributory degreesβ2(m, n) and β3(m, n), the synthesis means 114 weights the correspondingareas A2(m, n) and A3(m, n) and adds the weighted areas to area A1(m,n), thereby acquiring a processed frame Fr_(G).

FIG. 37 shows how similarities are computed in accordance with the ninthembodiment. As illustrated in the figure, the similarity computationmeans 112 partitions frame Fr₁ into m×n block-shaped areas A1(m, n) andperforms the parallel movement or affine transformation of each of theareas A1(m, n) with respect to frame Fr₂ and frame Fr₃. Further, areasin frames Fr₂ and Fr₃, in which a correlation between a pixel value inarea A(m, n) and a pixel value in frame Fr₂ or Fr₃ is highest, aredetected as corresponding areas A2(m, n) and A3(m, n) by the similaritycomputation means 112. When the correlation between pixel values ishighest, the accumulation of the square of a difference between pixelvalues in area A1(m, n) and area A2(m, n) and square of a differencebetween pixel values in area A1(m, n) and area A3(m, n), or thereciprocal of the accumulation of absolute values, is computed assimilarities b2(m, n) and b3 (m, n). For instance, in FIG. 37, areas inframes Fr₂ and Fr₃, which include a subject Q0 included in frame Fr₁ andhave the same size as area A1(1, 1), are detected as corresponding areasA2(1, 1) and A3(1, 1).

The contributory degree computation means 113 computes contributorydegrees β2(m, n) and β3(m, n), which are employed in weighing thecorresponding areas A2(m, n) and A3(m, n) and adding to the area A1(m,n), by multiplying similarities b2(m, n) and b3(m, n) by a predeterminedreference contributory degree k.

The synthesis means 114 acquires a processed frame Fr_(G) by weightingthe corresponding areas A2(m, n) and A3(m, n) and adding to the areaA1(m, n), in accordance with contributory degrees β2(m, n) and β3(m, n).More specifically, if frame data representing area A1(m, n) andcorresponding areas A2(m, n) and A3(m, n) are S1 (m, n), S2(m, n), andS3(m, n), and processed frame data representing an area (processed area)corresponding to area A1(m, n) in a processed frame Fr_(G) is SG(m, n),the processed frame data SG(m, n) is computed by the following Formula26.

SG(m, n)=S1(m, n)+β2(m, n)·S2(m, n)+β3(m, n)·S3(m, n)   (26)

Now, a description will be given of operation of the ninth embodiment.FIG. 38 shows processes that are performed in the ninth embodiment.First, the sampling means 101 samples frames Fr1, Fr2, and Fr3 fromvideo image data M0 (step S141). Then, in the similarity computationmeans 112, similarities b2(m, n) and b3(m, n) between area A1(m, n) inframe Fr1 and corresponding areas A2(m, n) and A3(m, n) are computed(step S142). Next, in the contributory degree computation means 113,contributory degrees β2(m, n) and β3(m, n) are computed by multiplyingsimilarities b2(m, n) and b3(m, n) by a reference contributory degree k(step S143). In accordance with contributory degrees β2(m, n) and β3(m,n), corresponding areas A2(m, n) and A3(m, n) are weighted and added toarea A1(m, n), whereby a processed frame FrG is obtained (step S144) andthe processing ends.

Thus, in the ninth embodiment, frame Fr1 is partitioned into a pluralityof areas A1(m, n), and similarities b2(m, n) and b3(m, n) are computedfor area A2(m, n) and area A3(m, n) in frames Fr2 and Fr3 whichcorrespond to area A1(m, n). If similarities b2(m, n) and b3(m, n) aregreat, contributory degrees (weighting coefficients) β2(m, n) and β3(m,n) are made greater. Corresponding areas A2(m, n) and area A3(m, n) areweighted and added to area A1(m, n), whereby a processed frame FrG isobtained. Because of this, even when a certain area in a video image ismoved, blurring can be removed for each area moved. As a result, aprocessed frame FrG with high quality can be obtained.

In the above-described ninth embodiment, although a processed frameFr_(G) is obtained by multiplying the corresponding areas A2(m, n) andA3(m, n) of frames Fr₂ and Fr₃ by contributory degrees β2(m, n) andβ3(m, n) and adding the weighted areas to area A1(m, n), a processedframe Fr_(G) with higher resolution than frame Fr₁ may be obtained byinterpolating the areas A2(m, n) and A3(m, n) multiplied by contributorydegrees β2(m, n) and β3(m, n) in area A1(m, n), like a method disclosedin Japanese Unexamined Patent Publication No. 2000-354244, for example.

Now, a description will be given of a tenth embodiment of the presentinvention. FIG. 39 shows an image processor constructed in accordancewith the tenth embodiment of the present invention. In the tenthembodiment, the same reference numerals will be applied to the sameparts as the eighth embodiment, so a detailed description of the sameparts will not be given. As illustrated in FIG. 39, the image processorof the tenth embodiment is equipped with motion-vector computation means105 and histogram processing means 106. The motion-vector computationmeans 105 partitions frame Fr₁ into m×n areas A1(m, n) and computes amotion vector V0(m, n) that represents the moving direction and movedquantity of area A1(m, n), for each area A1(m, n). The histogramprocessing means 106 computes a histogram H0, in which the magnitude ofmotion vector V0(m, n) is represented in the horizontal axis and thenumber of motion vectors V0(m, n) is represented in the vertical axis.Further, based on peaks in histogram H0, areas A1(m, n) are grouped foreach subject corresponding to the motion, and frame Fr₁ is partitionedinto a plurality of subject areas (e.g., O1 and O2 in this embodiment).

The image processor of the tenth embodiment is further equipped withsimilarity computation means 122, contributory degree computation means123, and synthesis means 124, instead of the similarity computationmeans 102, contributory degree computation means 103, and synthesismeans 104 of the eighth embodiment. The similarity computation means 122computes similarities b2(O1), b2(O2), b3(O1), and b3(O2) for subjectareas O1(Fr₂), O2(Fr₂), O1(Fr₃), and O2(Fr₃) in frames Fr₂ and Fr₃ whichcorrespond to the subject areas O1(Fr₁) and O2(Fr₁) of frame Fr₁. Thecontributory degree computation means 123 computes contributory degreesβ2(O1), β2(O2), β3(O1), and β3(O2) for subject areas O1(Fr₂), O2(Fr₂),O1(Fr₃), and O2(Fr₃). In accordance with the computed contributorydegrees β2(O1), β2(O2), β3(O1), and β3(O2), the synthesis means 114weights the corresponding subject areas O1(Fr₂), O2(Fr₂), O1(Fr₃), andO2(Fr₃) and adds the weighted areas to subject areas O1(Fr₁), O2(Fr₁),thereby acquiring a processed frame Fr_(G).

FIG. 40 shows how motion vector V0(m, n) is computed in accordance withthe tenth embodiment. If either a motion vector between frames Fr₁ andFr₂ or a motion vector between frames Fr₁ and Fr₃ is computed, frame Fr₁can be partitioned into a plurality of subject areas, so only thecomputation of a motion vector between frames Fr₁ and Fr₂ will bedescribed.

As illustrated in FIG. 40, the motion-vector computation means 105partitions frame Fr1 into m×n block-shaped areas A1(m, n) and moves eachof the areas A(m, n) in parallel with frame Fr1. When a correlationbetween pixel values in area A1(m, n) and frame Fr2 is highest, themoved quantity and moving direction of area A1(m, n) is computed asmotion vector V0(m, n) for that area A1(m, n). Note that when theaccumulation of the squares of differences between pixel values of areaA1(m, n) and frame Fr2 or accumulation of absolute values is smallest,the correlation is judged to be highest.

Now, assume that as shown in FIG. 41A, only the face of a person inframe Fr₁ has moved from the lower left part of frame Fr₂ to the upperright part of frame Fr₂. In this case, the magnitude of motion vectorV0(m, n) becomes greater for 4 areas A1(1, 1), A1(2, 1), A1(1, 2), andA1(2, 2) in the case of frame Fr₁ shown in FIG. 41B and smaller forother areas. Therefore, if the magnitude |V0(m, n)| of motion vectorV0(m, n) is represented by a histogram H0, there are two peaks, as shownin FIG. 42. Peak P1 corresponds to the motion vector V12(m, n) of areasother than areas A1(1, 1), A1(2, 1), A1(1, 2), and A1(2, 2), while peakP2 corresponds to the motion vector V22(m, n) of areas A1(1, 1), A1(2,1), A1(1, 2), and A1(2, 2).

Therefore, a plurality of areas A1(m, n) are represented by a firstsubject area O1 having a motion vector close to motion vector V12(m, n)and a second subject area O2 having a motion vector close to motionvector V22(m, n), so frame Fr₁ can be partitioned into two subject areasO1 and O2.

The similarity computation means 122 moves the subject areas O1 and O2of frame Fr₁ in parallel with frames Fr₂ and Fr₃. Further, areas inframes Fr₂ and Fr₃, in which a correlation between pixel values insubject areas O1, O2 and frames Fr₂, Fr₃ is highest, are detected ascorresponding subject areas O1(Fr₂), O2(Fr₂), O1(Fr₃), and O2(Fr₃) bythe similarity computation means 122. When the correlation between pixelvalues is highest, the reciprocal of the square of a difference betweenpixel values in subject areas O1, O2 and corresponding subject areasO1(Fr₂), O2(Fr₂), and reciprocal of the square of a difference betweenpixel values in subject areas O1, O2 and corresponding subject areasO1(Fr₃), O2(Fr₃), or the reciprocals of the absolute values, arecomputed as similarities b2(O1), b2(O2) and similarities b3(O1), b3(O2),respectively.

The contributory degree computation means 123 computes contributorydegrees β2(O1) and β2(O2) (which are employed in weighing thecorresponding subject areas O1(Fr₂) and O2(Fr₂) of frame Fr₂ and addingto the subject areas O1 and O2) and contributory degrees β3(O1) andβ3(O2) (which are employed in weighing the corresponding subject areasO1(Fr₃) and O2(Fr₃) of frame Fr₃ and adding to the subject areas O1 andO2) by multiplying similarities b2(O1), b2(O2), b3(O1), and b3(O2) by apredetermined reference contributory degree k.

The synthesis means 124 acquires a processed frame Fr_(G) by weightingthe corresponding subject areas O1(Fr₂), O2(Fr₂), O1(Fr₃), and O2(Fr₃)and adding to the subject areas O1 and O2, in accordance withcontributory degrees β2(O1), β2(O2), β3(O1), and β3(O2). Morespecifically, if frame data representing the subject areas O1, O2 andcorresponding areas O1(Fr₂), O2(Fr₂), O1(Fr₃), and O2(Fr₃) are SO1, SO2,SO1(Fr₂), SO2(Fr₂), SO1(Fr₃), and SO2(Fr₃), and processed frame datarepresenting subject areas (processed areas) of a processed frame Fr_(G)are SG1 and SG2, the processed frame data SG is computed by thefollowing Formula 27.

SG1=SO1+β2(O1)·SO1(Fr ₂)+β3(O1)·SO1(Fr ₃)

SG2=SO2+β2(O2)·SO2(Fr ₂)+β3(O2)·SO2(Fr ₃)   (27)

Now, a description will be given of operation of the tenth embodiment.FIG. 42 shows processes that are performed in the tenth embodiment.First, the sampling means 101 samples frames Fr₁, Fr₂, and Fr₃ fromvideo image data M0 (step S151). Then, in the motion-vector computationmeans 105, a plurality of motion vectors VO(m, n) are computed for theareas A1(m, n) of frame Fr₁ (step S152). Next, in the histogramprocessing means 106, histogram H0 is computed for motion vectors VO(m,n) (step S153). The areas A1(m, n) are grouped according to histogramH0, whereby frame Fr₁ is partitioned into subject areas O1 and O2 (stepS154).

Next, in the similarity computation means 122, similarities b2(O1) andb2(O2) between subject areas O1, O2 in frame Fr1 and correspondingsubject areas O1(Fr2) and O2(Fr2) in frame Fr2 are computed andsimilarities b3(O1) and b3(O2) between subject areas O1, O2 in frame Fr1and corresponding areas O1(Fr3) and O2(Fr3) in frame Fr3 are computed(step S155). Next, in the contributory degree computation means 123,contributory degrees β2(O1), β2(O2), β3(O1), and β3(O2) are computed bymultiplying similarities b2(O1), b2(O2) and b3(O1), and b3(O2) by areference contributory degree k (step S156). In accordance withcontributory degrees β2(O1) and β2(O2) and contributory degrees β3(O1)and β3(O2), the corresponding subject areas O1(Fr2) and O2(Fr2) andcorresponding subject areas O1(Fr3) and O2(Fr3) are weighted and addedto the subject areas O1 and O2, respectively. In this manner, aprocessed frame FrG is obtained (step S157) and the processing ends.

Thus, in the tenth embodiment, frame Fr1 is partitioned into a pluralityof subject areas O1 and O2, and similarities b2(O1) and b2(O2) andsimilarities b3(O1) and b3(O2) are computed for the subject areasO1(Fr2) and O2(Fr2) and subject areas O1(Fr3) and O2(Fr3) in frames Fr2and Fr3 which correspond to the subject areas O1 and O2. If similaritiesb2(O1) and b2(O2) and similarities b3(O1) and b3(O2) are great,contributory degrees (weighting coefficients) β2(O1), β2(O2), β3(O1),β3(O2) are made greater. The corresponding subject areas O1(Fr2) andO2(Fr2) and corresponding subject areas O1(Fr3) and O2(Fr3) are weightedand added to the subject areas O1 and O2, whereby a processed frame FrGis obtained. Because of this, even when a certain subject area in avideo image is moved, blurring can be removed for the subject areamoved. As a result, a processed frame FrG with higher quality can beobtained.

In the above-described tenth embodiment, although a processed frameFr_(G) is obtained by multiplying the corresponding subject areasO1(Fr₂) and O2(Fr₂) and corresponding subject areas O1(Fr₃) and O2(Fr₃)by contributory degrees β2(O1) and β2(O2) and contributory degreesβ3(O1) and β3(O2) and adding the weighted areas to the subject areas O1and O2, a processed frame Fr_(G) with higher resolution than frame Fr₁may be obtained by interpolating the corresponding subject areasO1(Fr₂), O2(Fr₂), O1(Fr₃), and O2(Fr₃) multiplied by contributorydegrees β2(O1), β2(O2), β3(O1), and β3(O2) in the subject areas O1 andO2, like a method disclosed in Japanese Unexamined Patent PublicationNo. 2000-354244, for example.

While the present invention has been described with reference to thepreferred embodiments thereof, the invention is not to be limited to thedetails given herein, but may be modified within the scope of theinvention hereinafter claimed.

1. A video image synthesis method comprising the steps of: sampling twocontiguous frames from a video image; placing a reference patchcomprising one or a plurality of rectangular areas on one of said twoframes which is used as a reference frame, then placing on the other ofsaid two frames a second patch which is the same as said referencepatch, then moving and/or deforming said second patch in said otherframe so that an image within said second patch coincides with an imagewithin said reference patch, and estimating a correspondent relationshipbetween a pixel within said second patch on said other frame and a pixelwithin said reference patch on said reference frame, based on saidsecond patch after the movement and/or deformation and on said referencepatch; acquiring a first interpolated frame whose resolution is higherthan each of said frames, by performing interpolation either on theimage within said second patch of said other frame or on the imagewithin said second patch of said other frame and image within saidreference patch of said reference frame, based on said correspondentrelationship; acquiring a second interpolated frame whose resolution ishigher than each of said frames, by performing interpolation on theimage within said reference patch of said reference frame; acquiring acoordinate-transformed frame by transforming coordinates of the imagewithin said second patch of said other frame to a coordinate space ofsaid reference frame, based on said correspondent relationship;computing a correlation value that represents a correlation between theimage within the patch of said coordinate-transformed frame and theimage within said reference patch of said reference frame; acquiring aweighting coefficient that makes a weight of said first interpolatedframe greater as said correlation becomes greater, when synthesizingsaid first interpolated frame and second interpolated frame, based onsaid correlation value; and acquiring a synthesized frame by weightingand synthesizing said first and second interpolated frames, based onsaid weighting coefficient.
 2. A video image synthesis method comprisingthe steps of: sampling three or more contiguous frames from a videoimage; placing a reference patch comprising one or a plurality ofrectangular areas on one of said three or more frames which is used as areference frame, then respectively placing on the others of said threeor more frames patches which are the same as said reference patch, thenmoving and/or deforming said patches in said other frames so that animage within the patch of each of said other frames coincides with animage within said reference patch, and respectively estimatingcorrespondent relationships between pixels within the patches of saidother frames and a pixel within said reference patch of said referenceframe, based on the patches of said other frames after the movementand/or deformation and on said reference patch; acquiring a plurality offirst interpolated frames whose resolution is higher than each of saidframes, by performing interpolation either on the image within the patchof each of said other frames or on the image within the patch of each ofsaid other frames and image within said reference patch of saidreference frame, based on said correspondent relationships; acquiringone or a plurality of second interpolated frames whose resolution ishigher than each of said frames and which are correlated with saidplurality of first interpolated frames, by performing interpolation onthe image within said reference patch of said reference frame; acquiringa plurality of coordinate-transformed frames by transforming coordinatesof the images within the patches of said other frames to a coordinatespace of said reference frame, based on said correspondentrelationships; computing correlation values that represent a correlationbetween the image within the patch of each of saidcoordinate-transformed frames and the image within said reference patchof said reference frame; acquiring weighting coefficients that make aweight of said first interpolated frame greater as said correlationbecomes greater, when synthesizing said first interpolated frame andsecond interpolated frame, based on said correlation values; andacquiring intermediate synthesized frames by weighting and synthesizingsaid first and second interpolated frames that correspond to each otheron the basis of said weighting coefficients, and acquiring a synthesizedframe by synthesizing said intermediate synthesized frames.
 3. Thesynthesis method as set forth in claim 1, wherein when said correlationvalue has been computed for each of the pixels and/or each local regionthat constitute each of said frames, said correlation value is filteredto compute a filtered correlation value, and said weighting coefficientis acquired based on said filtered correlation value.
 4. The synthesismethod as set forth in claim 2, wherein when said correlation value hasbeen computed for each of the pixels and/or each local region thatconstitute each of said frames, said correlation value is filtered tocompute a filtered correlation value, and said weighting coefficient isacquired based on said filtered correlation value.
 5. The synthesismethod as set forth in claim 1, wherein when said correlation value hasbeen computed for each of the pixels and/or each local region thatconstitute each of said frames, said weighting coefficient isinterpolated to acquire weighting coefficients for all pixels thatconstitute said first and second interpolated frames.
 6. The synthesismethod as set forth in claim 2, wherein when said correlation value hasbeen computed for each of the pixels and/or each local region thatconstitute each of said frames, said weighting coefficient isinterpolated to acquire weighting coefficients for all pixels thatconstitute said first and second interpolated frames.
 7. The synthesismethod as set forth in claim 1, wherein said weighting coefficient isacquired by referring to a nonlinear graph in which said correlationvalue is represented in the horizontal axis and said weightingcoefficient in the vertical axis.
 8. The synthesis method as set forthin claim 2, wherein said weighting coefficient is acquired by referringto a nonlinear graph in which said correlation value is represented inthe horizontal axis and said weighting coefficient in the vertical axis.9. The synthesis method as set forth in claim 1, wherein the estimationof said correspondent relationship, acquisition of said firstinterpolated frame, acquisition of said second interpolated frame,acquisition of said coordinate-transformed frame, computation of saidcorrelation value, acquisition of said weighting coefficient, andacquisition of said synthesized frame are performed by employing atleast one component that constitutes said frame.
 10. The synthesismethod as set forth in claim 2, wherein the estimation of saidcorrespondent relationship, acquisition of said first interpolatedframe, acquisition of said second interpolated frame, acquisition ofsaid coordinate-transformed frame, computation of said correlationvalue, acquisition of said weighting coefficient, and acquisition ofsaid synthesized frame are performed by employing at least one componentthat constitutes said frame.
 11. A video image synthesizer comprising:sampling means for sampling two contiguous frames from a video image;correspondent relationship estimation means for placing a referencepatch comprising one or a plurality of rectangular areas on one of saidtwo frames which is used as a reference frame, then placing on the otherof said two frames a second patch which is the same as said referencepatch, then moving and/or deforming said second patch in said otherframe so that an image within said second patch coincides with an imagewithin said reference patch, and estimating a correspondent relationshipbetween a pixel within said second patch on said other frame and a pixelwithin said reference patch on said reference frame, based on saidsecond patch after the movement and/or deformation and on said referencepatch; first interpolation means for acquiring a first interpolatedframe whose resolution is higher than each of said frames, by performinginterpolation either on the image within said second patch of said otherframe or on the image within said second patch of said other frame andimage within said reference patch of said reference frame, based on saidcorrespondent relationship; second interpolation means for acquiring asecond interpolated frame whose resolution is higher than each of saidframes, by performing interpolation on the image within said referencepatch of said reference frame; coordinate transformation means foracquiring a coordinate-transformed frame by transforming coordinates ofthe image within said second patch of said other frame to a coordinatespace of said reference frame, based on said correspondent relationship;correlation-value computation means for computing a correlation valuethat represents a correlation between the image within the patch of saidcoordinate-transformed frame and the image within said reference patchof said reference frame; weighting-coefficient acquisition means foracquiring a weighting coefficient that makes a weight of said firstinterpolated frame greater as said correlation becomes greater, whensynthesizing said first interpolated frame and second interpolatedframe, based on said correlation value; and synthesis means foracquiring a synthesized frame by weighting and synthesizing said firstand second interpolated frames, based on said weighting coefficient. 12.A video image synthesizer comprising: sampling means for sampling threeor more contiguous frames from a video image; correspondent relationshipestimation means for placing a reference patch comprising one or aplurality of rectangular areas on one of said three or more frames whichis used as a reference frame, then respectively placing on the others ofsaid three or more frames patches which are the same as said referencepatch, then moving and/or deforming said patches in said other frames sothat an image within the patch of each of said other frames coincideswith an image within said reference patch, and respectively estimatingcorrespondent relationships between pixels within the patches of saidother frames and a pixel within said reference patch of said referenceframe, based on the patches of said other frames after the movementand/or deformation and on said reference patch; first interpolationmeans for acquiring a plurality of first interpolated frames whoseresolution is higher than each of said frames, by performinginterpolation either on the image within the patch of each of said otherframes or on the image within the patch of each of said other frames andimage within said reference patch of said reference frame, based on saidcorrespondent relationships; second interpolation means for acquiringone or a plurality of second interpolated frames whose resolution ishigher than each of said frames and which are correlated with saidplurality of first interpolated frames, by performing interpolation onthe image within said reference patch of said reference frame;coordinate transformation means for acquiring a plurality ofcoordinate-transformed frames by transforming coordinates of the imageswithin the patches of said other frames to a coordinate space of saidreference frame, based on said correspondent relationships;correlation-value computation means for computing correlation valuesthat represent a correlation between the image within the patch of eachof said coordinate-transformed frames and the image within saidreference patch of said reference frame; weighting-coefficientacquisition means for acquiring weighting coefficients that make aweight of said first interpolated frame greater as said correlationbecomes greater, when synthesizing said first interpolated frame andsecond interpolated frame, based on said correlation values; andsynthesis means for acquiring intermediate synthesized frames byweighting and synthesizing said first and second interpolated framesthat correspond to each other on the basis of said weightingcoefficients, and acquiring a synthesized frame by synthesizing saidintermediate synthesized frames.
 13. The video image synthesizer as setforth in claim 11, wherein when said correlation value has been computedfor each of pixels and/or each local region that constitute each of saidframes, said synthesizer further comprises means for filtering saidcorrelation value to compute a filtered correlation value, and saidweighting-coefficient acquisition means acquires said weightingcoefficient, based on said filtered correlation value.
 14. The videoimage synthesizer as set forth in claim 12, wherein when saidcorrelation value has been computed for each of pixels and/or each localregion that constitute each of said frames, said synthesizer furthercomprises means for filtering said correlation value to compute afiltered correlation value, and said weighting-coefficient acquisitionmeans acquires said weighting coefficient, based on said filteredcorrelation value.
 15. The video image synthesizer as set forth in claim11, wherein when said correlation value has been computed for each ofthe pixels and/or each local region that constitute each of said frames,said weighting-coefficient acquisition means performs interpolation onsaid weighting coefficient, thereby acquiring weighting coefficients forall pixels that constitute said first and second interpolated frames.16. The video image synthesizer as set forth in claim 12, wherein whensaid correlation value has been computed for each of the pixels and/oreach local region that constitute each of said frames, saidweighting-coefficient acquisition means performs interpolation on saidweighting coefficient, thereby acquiring weighting coefficients for allpixels that constitute said first and second interpolated frames. 17.The video image synthesizer asset forth in claim 11, wherein saidweighting-coefficient acquisition means acquires said weightingcoefficient by referring to a nonlinear graph in which said correlationvalue is represented in the horizontal axis and said weightingcoefficient in the vertical axis.
 18. The video image synthesizer assetforth in claim 12, wherein said weighting-coefficient acquisition meansacquires said weighting coefficient by referring to a nonlinear graph inwhich said correlation value is represented in the horizontal axis andsaid weighting coefficient in the vertical axis.
 19. The video imagesynthesizer as set forth in claim 11, wherein said correspondentrelationship estimation means, said first interpolation means, saidsecond interpolation means, said coordinate transformation means, saidcorrelation-value computation means, said weighting-coefficientacquisition means, and said synthesis means perform the estimation ofsaid correspondent relationship, acquisition of said first interpolatedframe, acquisition of said second interpolated frame, acquisition ofsaid coordinate-transformed frame, computation of said correlationvalue, acquisition of said weighting coefficient, and acquisition ofsaid synthesized frame, by employing at least one component thatconstitutes said frame.
 20. The video image synthesizer asset forth inclaim 12, wherein said correspondent relationship estimation means, saidfirst interpolation means, said second interpolation means, saidcoordinate transformation means, said correlation-value computationmeans, said weighting-coefficient acquisition means, and said synthesismeans perform the estimation of said correspondent relationship,acquisition of said first interpolated frame, acquisition of said secondinterpolated frame, acquisition of said coordinate-transformed frame,computation of said correlation value, acquisition of said weightingcoefficient, and acquisition of said synthesized frame, by employing atleast one component that constitutes said frame.
 21. A program forcausing a computer to execute a video image synthesis method comprising:a procedure of sampling two contiguous frames from a video image; aprocedure of placing a reference patch comprising one or a plurality ofrectangular areas on one of said two frames which is used as a referenceframe, then placing on the other of said two frames a second patch whichis the same as said reference patch, then moving and/or deforming saidsecond patch in said other frame so that an image within said secondpatch coincides with an image within said reference patch, andestimating a correspondent relationship between a pixel within saidsecond patch on said other frame and a pixel within said reference patchon said reference frame, based on said second patch after the movementand/or deformation and on said reference patch; a procedure of acquiringa first interpolated frame whose resolution is higher than each of saidframes, by performing interpolation either on the image within saidsecond patch of said other frame or on the image within said secondpatch of said other frame and image within said reference patch of saidreference frame, based on said correspondent relationship; a procedureof acquiring a second interpolated frame whose resolution is higher thaneach of said frames, by performing interpolation on the image withinsaid reference patch of said reference frame; a procedure of acquiring acoordinate-transformed frame by transforming coordinates of the imagewithin said second patch of said other frame to a coordinate space ofsaid reference frame, based on said correspondent relationship; aprocedure of computing a correlation value that represents a correlationbetween the image within the patch of said coordinate-transformed frameand the image within said reference patch of said reference frame; aprocedure of acquiring a weighting coefficient that makes a weight ofsaid first interpolated frame greater as said correlation becomesgreater, when synthesizing said first interpolated frame and secondinterpolated frame, based on said correlation value; and a procedure ofacquiring a synthesized frame by weighting and synthesizing said firstand second interpolated frames, based on said weighting coefficient. 22.A program for causing a computer to execute a video image synthesismethod comprising: a procedure of sampling three or more contiguousframes from a video image; a procedure of placing a reference patchcomprising one or a plurality of rectangular areas on one of said threeor more frames which is used as a reference frame, then respectivelyplacing on the others of said three or more frames patches which are thesame as said reference patch, then moving and/or deforming said patchesin said other frames so that an image within the patch of each of saidother frames coincides with an image within said reference patch, andrespectively estimating correspondent relationships between pixelswithin the patches of said other frames and a pixel within saidreference patch of said reference frame, based on the patches of saidother frames after the movement and/or deformation and on said referencepatch; a procedure of acquiring a plurality of first interpolated frameswhose resolution is higher than each of said frames, by performinginterpolation either on the image within the patch of each of said otherframes or on the image within the patch of each of said other frames andimage within said reference patch of said reference frame, based on saidcorrespondent relationships; a procedure of acquiring one or a pluralityof second interpolated frames whose resolution is higher than each ofsaid frames and which are correlated with said plurality of firstinterpolated frames, by performing interpolation on the image withinsaid reference patch of said reference frame; a procedure of acquiring aplurality of coordinate-transformed frames by transforming coordinatesof the images within the patches of said other frames to a coordinatespace of said reference frame, based on said correspondentrelationships; a procedure of computing correlation values thatrepresent a correlation between the image within the patch of each ofsaid coordinate-transformed frames and the image within said referencepatch of said reference frame; a procedure of acquiring weightingcoefficients that make a weight of said first interpolated frame greateras said correlation becomes greater, when synthesizing said firstinterpolated frame and second interpolated frame, based on saidcorrelation values; and a procedure of acquiring intermediatesynthesized frames by weighting and synthesizing said first and secondinterpolated frames that correspond to each other on the basis of saidweighting coefficients, and acquiring a synthesized frame bysynthesizing said intermediate synthesized frames.
 23. The program asset forth in claim 21, wherein when said correlation value has beencomputed for each pixel and/or each local region that constitutes eachof said frames, said method further comprises a procedure of filteringsaid correlation value to compute a filtered correlation value, and saidweighting-coefficient acquisition procedure is a procedure of acquiringsaid weighting coefficient, based on said filtered correlation value.24. The program as set forth in claim 22, wherein when said correlationvalue has been computed for each pixel and/or each local region thatconstitute each of said frames, said method further comprises aprocedure of filtering said correlation value to compute a filteredcorrelation value, and said weighting-coefficient acquisition procedureis a procedure of acquiring said weighting coefficients, based on saidfiltered correlation value.
 25. The program as set forth in claim 21,wherein when said correlation value has been computed for each of thepixels and/or each local region that constitute each of said frames,said weighting-coefficient acquisition procedure is a procedure ofperforming interpolation on said weighting coefficient and acquiringweighting coefficients for all pixels that constitute said first andsecond interpolated frames.
 26. The program as set forth in claim 22,wherein when said correlation value has been computed for each of thepixels and/or each local region that constitute each of said frames,said weighting-coefficient acquisition procedure is a procedure ofperforming interpolation on said weighting coefficients and acquiringweighting coefficients for all pixels that constitute said first andsecond interpolated frames.
 27. The program as set forth in claim 21,wherein said weighting-coefficient acquisition procedure is a procedureof acquiring said weighting coefficient by referring to a nonlineargraph in which said correlation value is represented in the horizontalaxis and said weighting coefficient in the vertical axis.
 28. Theprogram as set forth in claim 22, wherein said weighting-coefficientacquisition procedure is a procedure of acquiring said weightingcoefficients by referring to a nonlinear graph in which said correlationvalue is represented in the horizontal axis and said weightingcoefficient in the vertical axis.
 29. The program as set forth in claim21, wherein said correspondent relationship estimation procedure, saidfirst interpolated frame acquisition procedure, said second interpolatedframe acquisition procedure, said coordinate-transformed frameacquisition procedure, said correlation value computation procedure,said weighting-coefficient acquisition procedure, and saidsynthesized-frame acquisition procedure are procedures of performing theestimation of said correspondent relationship, acquisition of said firstinterpolated frame, acquisition of said second interpolated frame,acquisition of said coordinate-transformed frame, computation of saidcorrelation value, acquisition of said weighting coefficient, andacquisition of said synthesized frame by employing at least onecomponent that constitutes said frame, respectively.
 30. The program asset forth in claim 22, wherein said correspondent relationshipestimation procedure, said first interpolated frame acquisitionprocedure, said second interpolated frame acquisition procedure, saidcoordinate-transformed frame acquisition procedure, said correlationvalue computation procedure, said weighting-coefficient acquisitionprocedure, and said synthesized-frame acquisition procedure areprocedures of performing the estimation of said correspondentrelationship, acquisition of said first interpolated frame, acquisitionof said second interpolated frame, acquisition of saidcoordinate-transformed frame, computation of said correlation value,acquisition of said weighting coefficient, and acquisition of saidsynthesized frame by employing at least one component that constitutessaid frame, respectively.